def convert_tokens_to_tensors(batch: dict,
                              tokenize_labels: bool) -> dict[str, Any]:
    """HF Datasets converts tensors into lists when we store them, and we don't.

    want to use `type='torch'` because some content in the dataset, like
    generation args or single ints, should not be converted.

    Here, we convert those lists of tokens back into tensors in order to feed them into the model.

    Args:
        batch (dict): A dictionary of batched inputs
        tokenize_labels (bool): Whether or not the labels are tokenized (and need to be stacked)

    Returns:
        dict: The batch with torch tensors in the corresponding keys instead of lists of lists
    """
    batch['input_ids'] = torch.stack(
        list(map(torch.tensor, batch['input_ids'])),
    )
    if tokenize_labels:
        batch['labels'] = torch.stack(list(map(torch.tensor, batch['labels'])))
        batch['continuation_indices'] = list(
            map(torch.tensor, batch['continuation_indices']),
        )
    return batch
HF Datasets converts tensors into lists when we store them, and we don't.

    want to use `type='torch'` because some content in the dataset, like
    generation args or single ints, should not be converted.

    Here, we convert those lists of tokens back into tensors in order to feed them into the model.

    Args:
        batch (dict): A dictionary of batched inputs
        tokenize_labels (bool): Whether or not the labels are tokenized (and need to be stacked)

    Returns:
        dict: The batch with torch tensors in the corresponding keys instead of lists of lists
    
llmfoundry/eval/datasets/utils.py