def _ngrams_generator(reader):
    """A generator producing flattened and sanitized ngrams"""
    if reader.last_row_consumed:
        # This means that Dataset is trying to create a new instance of the generator. Do not
        # recommend to do that since this is an expensive operation. num_epochs is a more efficient way
        # to do this.
        warnings.warn(_RESET_READER_WARN, category=UserWarning)
        reader.reset()

    for next_sample in reader:
        yield _sanitize_and_flatten(next_sample)
A generator producing flattened and sanitized ngrams
petastorm/tf_utils.py