def num_tokens(self, index):
        """Return an example's length (number of tokens), used for batching."""
        # TODO make it configurable whether to use max() or sum() here
        return max(
            dataset.num_tokens(self._map_index(key, index))
            for key, dataset in self.datasets.items()
        )
Return an example's length (number of tokens), used for batching.
understand_icl/fairseq/fairseq/data/round_robin_zip_datasets.py