def get_grad_norm(params, scale=1):
    """Compute grad norm given a gradient scale."""
    total_norm = 0.0
    for p in params:
        if p.grad is not None:
            param_norm = (p.grad.detach().data / scale).norm(2)
            total_norm += param_norm.item() ** 2
    total_norm = total_norm**0.5
    return total_norm
Compute grad norm given a gradient scale.
dpkd/transformers/examples/pytorch/speech-pretraining/run_wav2vec2_pretraining_no_trainer.py