def tokenize(self, sql):
    """Tokenizes a SQL query into a list of SQL tokens."""
    return [str(token).strip() for token in \
              sqlparse.sql.TokenList(sqlparse.parse(sql)[0].tokens).flatten() \
              if str(token).strip()]
Tokenizes a SQL query into a list of SQL tokens.
language/xsp/data_preprocessing/sql_utils.py