def special_convert_tokens_to_ids(*args, **kwargs):
      """Maps all the added tokens to one of the unused tokens."""
      ids = self.tokenizer.convert_tokens_to_ids(*args, **kwargs)
      # size without added tokens
      original_vocab_size = self.tokenizer.vocab_size
      return [
          id_ if id_ < original_vocab_size else (id_ % original_vocab_size) + 1
          for id_ in ids
      ]  # +1 to skip the [PAD] token
Maps all the added tokens to one of the unused tokens.
language/search_agents/muzero/env.py