def tokenize(batch: Batch, *, tokenizer_dict: Dict[str,
                                                   TokenizerFn]) -> jnp.ndarray:
  """Tokenize video into discrete codes by the vq_model encoder."""
  return tokenizer_dict['tokenizer'](batch['inputs'])
Tokenize video into discrete codes by the vq_model encoder.
videogvt/interfaces/mm_vq.py