def print_rank_last(message):
    """If distributed is initialized, print only on last rank."""
    if torch.distributed.is_initialized():
        if is_last_rank():
            print(message, flush=True)
    else:
        print(message, flush=True)
If distributed is initialized, print only on last rank.
training/DeepSpeed-Domino/megatron/utils.py