def lmx_moe():
  """Transformer with mixture of experts.  140M Params."""
  hparams = lmx_base()
  hparams.ffn_layer = "local_moe_tpu"
  return hparams
Transformer with mixture of experts.  140M Params.
tensor2tensor/models/research/lm_experiments.py