def check_uniques(example, uniques):
    """Check if current hash is still in set of unique hashes and remove if true."""
    if example["hash"] in uniques:
        uniques.remove(example["hash"])
        return True
    else:
        return False
Check if current hash is still in set of unique hashes and remove if true.
dpkd/transformers/examples/research_projects/codeparrot/scripts/preprocessing.py