def token_to_char_ids_mapper(keys_to_map, word_length=50, suffix="_cid"):
  """Creates a mapping function to augment a tf.data.Dataset with character ids.

  Suppose we have a `dataset` with outputs `str1` and `str2` of arbitrary shape.
  Here is an example usage of this function:

  dataset = dataset.map(string_to_char_ids(['str1', 'str2']))

  Now the dataset will also include outputs `str1_cid` and `str2_cid` that can
  be used as features in a model. The 'str1_cid' feature will be a int32 Tensor
  of shape str1.get_shape() + [word_length].

  Args:
    keys_to_map: List of strings that are keys for tf.string Tensors to map to
        character ids.
    word_length: Number of bytes to include per word.
    suffix: String to append to the given keys to indicate the mapped Tensors.

  Returns:
    _mapper: A mapping function that can be used with the tf.data.Dataset API.
  """
  def _mapper(dataset):
    for k in keys_to_map:
      dataset[k + suffix] = batch_word_to_char_ids(dataset[k], word_length)
    return dataset
  return _mapper
Creates a mapping function to augment a tf.data.Dataset with character ids.

  Suppose we have a `dataset` with outputs `str1` and `str2` of arbitrary shape.
  Here is an example usage of this function:

  dataset = dataset.map(string_to_char_ids(['str1', 'str2']))

  Now the dataset will also include outputs `str1_cid` and `str2_cid` that can
  be used as features in a model. The 'str1_cid' feature will be a int32 Tensor
  of shape str1.get_shape() + [word_length].

  Args:
    keys_to_map: List of strings that are keys for tf.string Tensors to map to
        character ids.
    word_length: Number of bytes to include per word.
    suffix: String to append to the given keys to indicate the mapped Tensors.

  Returns:
    _mapper: A mapping function that can be used with the tf.data.Dataset API.
  
language/common/inputs/char_utils.py