def whitespace_tokenize(text):
    """Runs basic whitespace cleaning and splitting on a piece of text."""
    tokens = []
    if not text:
        return tokens
    sents = split_text_into_sentences_by_symbol(text, include_symbol=True)
    for sent, idx in sents:
        tokens.extend(sent.split())
    return tokens
Runs basic whitespace cleaning and splitting on a piece of text.
pycorrector/utils/tokenizer.py