def attention_bias_lower_triangle(length):
  """Create a bias tensor to be added to attention logits.

  Masked elements will have an effective negative-infinity value.

  Args:
    length: Integer specifying maximum sequence length in batch.

  Returns:
    <float>[length, length]
  """
  # First, create a sequence mask, e.g.:
  # [1, 0, ..., 0]
  # ...
  # [1, 1, ..., 1]
  sequence_mask = tf.sequence_mask(tf.range(1, length + 1), length)
  # Invert to transform to attention biases.
  attention_bias = tf.to_float(tf.logical_not(sequence_mask)) * -1e9
  return attention_bias
Create a bias tensor to be added to attention logits.

  Masked elements will have an effective negative-infinity value.

  Args:
    length: Integer specifying maximum sequence length in batch.

  Returns:
    <float>[length, length]
  
language/xsp/model/common_layers.py