def preserve_token(token, vocab):
  """Returns True if the token should forgo tokenization and be preserved."""
  if not FLAGS.preserve_unused_tokens:
    return False
  if token not in vocab:
    return False
  return bool(_UNUSED_TOKEN_RE.search(token))
Returns True if the token should forgo tokenization and be preserved.
test/external/mlperf_bert/preprocessing/tokenization.py