def _get_token_idx(self, x):
        """
        This is to extend noising functions to be able to apply to non-bpe
        tokens, e.g. word or characters.
        """
        x = torch.t(x)
        word_idx = np.array([range(len(x_i)) for x_i in x])
        return np.transpose(word_idx)

        This is to extend noising functions to be able to apply to non-bpe
        tokens, e.g. word or characters.
        
structured_prompting/fairseq-version/fairseq/fairseq/data/noising.py