def whitespace_tokenize(text):
  """Runs basic whitespace cleaning and splitting on a piece of text."""
  text = text.strip()
  if not text:
    return []
  tokens = text.split()
  return tokens
Runs basic whitespace cleaning and splitting on a piece of text.
structured_prompting/fairseq-version/struprompting/data/squad/basic_tokenizer.py