def lazy_tokenizer(batch):
    """Tokenizes if there is something to tokenize."""

    if len(batch) > 0 and isinstance(batch[0], str):
        batch = [word_tokenize(utt) for utt in batch]
    return batch
Tokenizes if there is something to tokenize.
deeppavlov/models/tokenizers/lazy_tokenizer.py