def get_prefix(sentence, prefix_len):
    """assuming no bpe, gets the prefix of the sentence with prefix_len words"""
    tokens = sentence.strip("\n").split()
    if prefix_len >= len(tokens):
        return sentence.strip("\n")
    else:
        return " ".join(tokens[:prefix_len])
assuming no bpe, gets the prefix of the sentence with prefix_len words
structured_prompting/fairseq-version/fairseq/examples/noisychannel/rerank_utils.py