def _tokenize(self, text: str) -> List[str]:
        """Split a string into tokens using BPE."""
        split_tokens = []

        words = re.findall(r"\S+\n?", text)

        for token in words:
            split_tokens.extend(list(self.bpe(token).split(" ")))
        return split_tokens
Split a string into tokens using BPE.
dpkd/transformers/src/transformers/models/blenderbot_small/tokenization_blenderbot_small.py