def get_dataset_stats(dataset):
    """Get stats about dataset based on src_lengths of samples"""
    max_len = 0
    min_len = 100000
    avg_len = 0
    for d in dataset:
        max_len = max(max_len, d["net_input"]["src_lengths"].item())
        min_len = min(min_len, d["net_input"]["src_lengths"].item())
        avg_len += d["net_input"]["src_lengths"].item()

    return max_len, min_len, avg_len / len(dataset)
Get stats about dataset based on src_lengths of samples
structured_prompting/fairseq-version/fairseq/examples/speech_to_speech/benchmarking/data_utils.py