def entropy(x):
    """Calculate entropy of a pre-softmax logit Tensor"""
    exp_x = torch.exp(x)
    A = torch.sum(exp_x, dim=1)  # sum of exp(x_i)
    B = torch.sum(x * exp_x, dim=1)  # sum of x_i * exp(x_i)
    return torch.log(A) - B / A
Calculate entropy of a pre-softmax logit Tensor
dpkd/transformers/examples/research_projects/deebert/src/modeling_highway_bert.py