def _separate_chunks_by_size(initial_chunks: list) -> tuple[list[str], list[tuple]]:
    """Separate chunks into small and large based on token count."""
    small_chunks = []
    large_chunks_data = []

    for i, chunk in enumerate(initial_chunks):
        if chunk.token_count <= SAFE_CHUNK_SIZE:
            small_chunks.append(chunk.text)
        else:
            large_chunks_data.append((i, chunk))

    return small_chunks, large_chunks_data
Separate chunks into small and large based on token count.
backend/airweave/platform/transformers/default_file_chunker.py