def read_file(filename):
  """Read the contents of filename (str) and split into documents by chapter."""

  all_documents = []
  # Input file format:
  # See wiki.*.tokens for an example
  # Documents are contiguously stored. Headers to sections are denoted by
  # "=" on each side. The number of "=" indicates the depth. So the title for
  # United States is "= United States =". "= = = Population = = =" indicates
  # a section 3 levels deep (United States -> Demographics -> Population).
  # There are blank lines between sections.
  # Each line is a paragraph. Periods that are sentence delimiters have a space
  # on each side. Periods in abbreviations do not have spaces.

  # For parallel processing, we first split the single file into documents.
  # Each document is a list of lines.
  with tf.gfile.GFile(filename, "r") as reader:
    for line in reader:
      line = line.strip()
      if not line:
        continue

      # Headers like (" = John Doe = ") are document delimiters
      if line.startswith("= ") and line[2] != "=":
        all_documents.append([])
      # Skip lines that are headers
      if line[0] == "=":
        continue
      all_documents[-1].append(line)

  # Remove empty documents
  all_documents = [x for x in all_documents if x]
  return all_documents
Read the contents of filename (str) and split into documents by chapter.
language/conpono/create_pretrain_data/wiki_preproc_pipeline.py