def tokens_to_features_dict(tokens, name_prefix):
  """Converts a list of Tokens into a dict of TF Features."""
  as_tuples = [(t.text.encode(), t.start, t.stop, t.id) for t in tokens]
  features_dict = {}
  for i, (name_suffix, feature_type) in enumerate([('texts', bytes_feature),
                                                   ('starts', int_feature),
                                                   ('stops', int_feature),
                                                   ('ids', int_feature)]):
    feature_name = '{}_token_{}'.format(name_prefix, name_suffix)
    features_dict[feature_name] = feature_type([x[i] for x in as_tuples])
  return features_dict
Converts a list of Tokens into a dict of TF Features.
language/realm/generate_retrieval_corpus.py