def generate_samples(self, data_dir, tmp_dir, data_split):
    """This function generates train and validation pairs in t2t-datagen style.

    The function assumes that if you have data at one level of the pipeline,
    you don't want to re-generate it, so for example if the 4 txt files exist,
    the function continues by generating the t2t-datagen format files.
    So if you want to re-download or re-generate data,
    you have to delete it first from the appropriate directories.

    Args:
      data_dir: string, Directory where the data will be generated. The raw
                        data has to be downloaded one directory level higher.
      tmp_dir: string, temp directory.
      data_split: string, which data split to generate samples for

    Yields:
      dict
    """

    self.data_dir = data_dir
    print('problem_log: ' +
          self.mode[data_split] + ' data generation activated.')

    s_path = os.path.join(data_dir, self.mode[data_split] + 'Source.txt')
    t_path = os.path.join(data_dir, self.mode[data_split] + 'Target.txt')

    # Open the files and yield source-target lines.
    with tf.gfile.GFile(s_path, mode='r') as source_file:
      with tf.gfile.GFile(t_path, mode='r') as target_file:
        source, target = source_file.readline(), target_file.readline()
        while source and target:
          yield {'inputs': source.strip(), 'targets': target.strip()}
          source, target = source_file.readline(), target_file.readline()
This function generates train and validation pairs in t2t-datagen style.

    The function assumes that if you have data at one level of the pipeline,
    you don't want to re-generate it, so for example if the 4 txt files exist,
    the function continues by generating the t2t-datagen format files.
    So if you want to re-download or re-generate data,
    you have to delete it first from the appropriate directories.

    Args:
      data_dir: string, Directory where the data will be generated. The raw
                        data has to be downloaded one directory level higher.
      tmp_dir: string, temp directory.
      data_split: string, which data split to generate samples for

    Yields:
      dict
    
tensor2tensor/data_generators/dialog_abstract.py