SentenceTransformer

This model was finetuned with Unsloth.

based on unsloth/embeddinggemma-300m

This is a sentence-transformers model finetuned from unsloth/embeddinggemma-300m. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: unsloth/embeddinggemma-300m
Maximum Sequence Length: 512 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'PeftModelForFeatureExtraction'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
  (3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
  (4): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    "TITLE: Essentialism: The Disciplined Pursuit of Less\nGENRES: business, fiction, nonfiction, philosophy, self-help\nAUTHORS: Greg McKeown\nDESCRIPTION: Have you ever found yourself stretched too thin? Do you simultaneously feel overworked and underutilized? Are you often busy but not productive? Do you feel like your time is constantly being hijacked by other people's agendas? If you answered yes to any of these, the way out is the Way of the Essentialist. The Way of the Essentialist isn't about getting more done in less time. It's about getting only the right thingsdone. It is not a time management strategy, or a productivity technique. It is a systematic disciplinefor discerning what is absolutely essential, then eliminating everything that is not, so we can make the highest possible contribution towards the things that really matter. By forcing us to apply a more selective criteria for what is Essential, the disciplined pursuit of less empowers us to reclaim control of our own choices about where to spend our precious time and energy - instead of giving others the implicit permission to choose for us. Essentialism is not one more thing - it's a whole new way of doing everything. A must-read for any leader, manager, or individual who wants to learn who to do less, but better, in every area of their lives, Essentialism is a movement whose time has come.",
    'TITLE: The One Thing: The Surprisingly Simple Truth Behind Extraordinary Results\nGENRES: business, fiction, nonfiction, philosophy, self-help\nAUTHORS: Gary Keller, Jay Papasan\nDESCRIPTION: The One Thingexplains the success habit to overcome the six lies that block our success, beat the seven thieves that steal time, and leverage the laws of purpose, priority, and productivity.',
    'TITLE: Smarter than Squirrels\nGENRES: adventure, children, fantasy, fiction, middle grade, young adult\nAUTHORS: Lucy Nolan, Mike Reed\nDESCRIPTION: THE HILARIOUS ADVENTURES OF TWO CONFUSED CANINES Down Girl and Sit are two dogs who are "smarter than squirrels." They know how to protect their masters from all the things that can go wrong in the neighborhood: they bark at paperboys and guard the garbage cans, and keep mischievous squirrels at bay. But when Here Kitty Kitty moves in next door, their daily routines are turned topsy-turvy. Filled with humor and adventure, this illustrated chapter book takes a look at life in the backyard from the well-intentioned but misguided viewpoint of man\'s best friend.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0010, 0.8911, 0.1610],
#         [0.8911, 1.0010, 0.2207],
#         [0.1610, 0.2207, 1.0000]], dtype=torch.float16)

Training Details

Training Dataset

Unnamed Dataset

Size: 19,941 training samples
Columns: anchor_text and positive_text
Approximate statistics based on the first 1000 samples:
anchor_text positive_text
type string string
details
min: 36 tokens
mean: 245.14 tokens
max: 512 tokens

min: 42 tokens
mean: 241.58 tokens
max: 512 tokens

	anchor_text	positive_text
type	string	string
details	min: 36 tokens mean: 245.14 tokens max: 512 tokens	min: 42 tokens mean: 241.58 tokens max: 512 tokens

Samples:

anchor_text	positive_text
TITLE: The View from the Cheap Seats: Selected Nonfiction GENRES: biography, contemporary, fantasy, fiction, horror, memoir, nonfiction, philosophy, science, science fiction AUTHORS: Neil Gaiman DESCRIPTION: An enthralling collection of nonfiction essays on myriad topics--from art and artists to dreams, myths, and memories--observed in Neil Gaiman's probing, amusing, and distinctive style. An inquisitive observer, thoughtful commentator, and assiduous craftsman, Neil Gaiman has long been celebrated for the sharp intellect and startling imagination that informs his bestselling fiction. Now, The View from the Cheap Seats brings together for the first time ever more than sixty pieces of his outstanding nonfiction. Analytical yet playful, erudite yet accessible, this cornucopia explores a broad range of interests and topics, including (but not limited to): authors past and present; music; storytelling; comics; bookshops; travel; fairy tales; America; inspiration; libraries; ghosts; and the...	TITLE: The Art of Asking; or, How I Learned to Stop Worrying and Let People Help GENRES: biography, business, contemporary, fiction, memoir, nonfiction, philosophy, self-help AUTHORS: Amanda Palmer DESCRIPTION: Rock star, crowdfunding pioneer, and TED speaker Amanda Palmer knows all about asking. Performing as a living statue in a wedding dress, she wordlessly asked thousands of passersby for their dollars. When she became a singer, songwriter, and musician, she was not afraid to ask her audience to support her as she surfed the crowd (and slept on their couches while touring). And when she left her record label to strike out on her own, she asked her fans to support her in making an album, leading to the world's most successful music Kickstarter. Even while Amanda is both celebrated and attacked for her fearlessness in asking for help, she finds that there are important things she cannot ask for-as a musician, as a friend, and as a wife. She learns that she isn't alone in this, that s...
TITLE: Styxx (Dark-Hunter, #22) GENRES: adventure, contemporary, fantasy, fiction, horror, paranormal, romance, science, science fiction, urban fantasy AUTHORS: Sherrilyn Kenyon DESCRIPTION: Just when you thought doomsday was over... Centuries ago Acheron saved the human race by imprisoning an ancient evil bent on absolute destruction. Now that evil has been unleashed and it is out for revenge. As the twin to Acheron, Styxx hasn't always been on his brother's side. They've spent more centuries going at each other's throats than protecting their backs. Now Styxx has a chance to prove his loyalty to his brother, but only if he's willing to trade his life and future for Acheron's. The Atlantean goddess of Wrath and Misery, Bethany was born to right wrongs. But it was never a task she relished. Until now. She owes Acheron a debt that she vows to repay, no matter what it takes. He will join their fellow gods in hell and nothing is going to stop her. But things are never what they seem, and ...	TITLE: Dark Skye (Immortals After Dark, #15) GENRES: adventure, contemporary, fantasy, fiction, paranormal, romance, urban fantasy AUTHORS: Kresley Cole DESCRIPTION: In this highly anticipated fifteenth novel in the Immortals After Darkseries, #1 New York Timesbestselling author Kresley Cole spins a sultry tale of a mighty warrior scarred inside and out and the beguiling sorceress with the power to heal him--or vanquish him forever. ETERNAL OBSESSION As a boy, Thronos, prince of Skye Hall, loved Lanthe, a mischievous Sorceri girl who made him question everything about his Vrekener clan. But when the two got caught in the middle of their families' war, tragedy struck, leaving Thronos and Lanthe bitter enemies. Though centuries have passed, nothing can cool his seething need for the beautiful enchantress who scarred his body--and left an even deeper impression on his soul. ENDLESS YEARNING Lanthe, a once-formidable sorceress struggling to reclaim her gifts, searches for love and acceptan...
TITLE: Marked (Dark Protectors, #7) GENRES: contemporary, fantasy, fiction, paranormal, romance, science fiction, urban fantasy AUTHORS: Rebecca Zanetti DESCRIPTION: Janie Kayrs has known Zane almost her whole life. He was her friend in the dream world. She trusted him. But that was before he kidnapped her, spiriting her away to an isolated cabin to learn what her dreams never told her. Like how dangerous he looks. How he got on the wrong side of the negotiating table. And how much sexier he is in real life... Zane is a battle-hardened warrior, used to command and solitude. But Janie has drawn him from the minute they met. His need for her could destroy everything he's worked for, but the risk is too sweet not to take it. They call her the Chosen One. But when it comes down to the questions of peace or war, life or death, safety or passion, it will be Janie who makes the choice...	TITLE: Dark Skye (Immortals After Dark, #15) GENRES: adventure, contemporary, fantasy, fiction, paranormal, romance, urban fantasy AUTHORS: Kresley Cole DESCRIPTION: In this highly anticipated fifteenth novel in the Immortals After Darkseries, #1 New York Timesbestselling author Kresley Cole spins a sultry tale of a mighty warrior scarred inside and out and the beguiling sorceress with the power to heal him--or vanquish him forever. ETERNAL OBSESSION As a boy, Thronos, prince of Skye Hall, loved Lanthe, a mischievous Sorceri girl who made him question everything about his Vrekener clan. But when the two got caught in the middle of their families' war, tragedy struck, leaving Thronos and Lanthe bitter enemies. Though centuries have passed, nothing can cool his seething need for the beautiful enchantress who scarred his body--and left an even deeper impression on his soul. ENDLESS YEARNING Lanthe, a once-formidable sorceress struggling to reclaim her gifts, searches for love and acceptan...

Loss: MultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "gather_across_devices": false
}

Evaluation Dataset

Unnamed Dataset

Size: 59 evaluation samples
Columns: anchor_text and positive_text
Approximate statistics based on the first 59 samples:
anchor_text positive_text
type string string
details
min: 35 tokens
mean: 218.02 tokens
max: 512 tokens

min: 79 tokens
mean: 233.36 tokens
max: 439 tokens

	anchor_text	positive_text
type	string	string
details	min: 35 tokens mean: 218.02 tokens max: 512 tokens	min: 79 tokens mean: 233.36 tokens max: 439 tokens

Samples:

anchor_text	positive_text
TITLE: White Pine GENRES: children, fiction, historical fiction, history, middle grade, young adult AUTHORS: Caroline Akervik DESCRIPTION: White Pine is an historical fiction set in the nineteenth century. After Sevy Anderson's father breaks his leg in a sawmill accident, the fourteen-year-old must take his place with the rough and tumble lumberjacks and river rats who harvest the white pine forests of Wisconsin. The men of the Northwoods live hard and on the edge, and Sevy must prove his courage and his worth in the company of legends. Will he become the man he so longs to be? Will the other men ever accept him? And will he even survive his first winter in the Northwoods?	TITLE: Pia the Penguin Fairy (Rainbow Magic: Ocean Fairies, #3) GENRES: children, fantasy, fiction, middle grade AUTHORS: Daisy Meadows, Georgie Ripper DESCRIPTION: The Ocean Fairies keep all the sea creatures safe and happy -- until their magic goes missing! This is our eleventh group of Rainbow Magic fairies. The Ocean Fairies keep all the sea creatures safe and happy! But when the goblins shatter their enchanted conch shell, seven magical sea creatures leave to search for the pieces. The Ocean Fairies must find the shells . . . and their animal friends! Rachel and Kirsty are off to the South Pole! Pia the Penguin Fairy's penguin, Scamp, is somewhere in the chilly waters --- but where? Find the missing creature in each book and help save the ocean magic!
TITLE: Skin Game (The Dresden Files, #15) GENRES: adventure, contemporary, crime, fantasy, fiction, horror, mystery, paranormal, romance, science, science fiction, thriller, urban fantasy AUTHORS: Jim Butcher DESCRIPTION: Harry Dresden, Chicago's only professional wizard, is about to have a very bad day.... Because as Winter Knight to the Queen of Air and Darkness, Harry never knows what the scheming Mab might want him to do. Usually, it's something awful. He doesn't know the half of it.... Mab has just traded Harry's skills to pay off one of her debts. And now he must help a group of supernatural villains--led by one of Harry's most dreaded and despised enemies, Nicodemus Archleone--to break into the highest-security vault in town so that they can then access the highest-security vault in the Nevernever. It's a smash-and-grab job to recover the literal Holy Grail from the vaults of the greatest treasure hoard in the supernatural world--which belongs to the one and only Hades, Lord of ...	TITLE: Hunted (The Iron Druid Chronicles, #6) GENRES: adventure, contemporary, fantasy, fiction, mystery, paranormal, romance, science, science fiction, thriller, urban fantasy, young adult AUTHORS: Kevin Hearne DESCRIPTION: For a two-thousand-year-old Druid, Atticus O'Sullivan is a pretty fast runner. Good thing, because he's being chased by not one but two goddesses of the hunt--Artemis and Diana--for messing with one of their own. Dodging their slings and arrows, Atticus, Granuaile, and his wolfhound Oberon are making a mad dash across modern-day Europe to seek help from a friend of the Tuatha De Danann. His usual magical option of shifting planes is blocked, so instead of playing hide-and-seek, the game plan is . . . run like hell. Crashing the pantheon marathon is the Norse god Loki. Killing Atticus is the only loose end he needs to tie up before unleashing Ragnarok--AKA the Apocalypse. Atticus and Granuaile have to outfox the Olympians and contain the god of mischief if they want...
TITLE: Slow Curve on the Coquihalla (A Hunter Rayne Highway Mystery, #1) GENRES: crime, fiction, mystery, thriller AUTHORS: R.E. Donald DESCRIPTION: When a well respected truck driver, the owner of a family trucking business, is found dead in his truck down a steep embankment along the Coquihalla highway that winds through the mountains in British Columbia, his distraught daughter wants to know how and why he died. Not long afterwards, while driving the same highway, her husband's brakes are tampered with, almost creating another fatal accident on a treacherous incline, This compels Hunter Rayne, a fellow trucker and retired RCMP detective, to help her find answers. As he uncovers signs of illegal cross border activity originating in a Seattle warehouse, Hunter recruits an old friend, an outlaw biker, to infiltrate what appears to be an international smuggling ring. But while Hunter follows up clues and waits for critical information from his old friend, the wily biker starts to play h...	TITLE: The Rock Star GENRES: crime, fiction, horror, mystery, romance, thriller AUTHORS: Rick Soper DESCRIPTION: Jake Martin saves rock star Jimi Christian from drowning on a California beach, and triggers a wave of unintended consequences... Injured FBI Special Agent Jon Stevens is back on the job, with a broken skull, a broken spirit and a ritualistic murder case only his talents can solve... And a vigilante begins targeting celebrities he deems worthy of his own savage brand of justice. What ties them all together? A revenge served so cold and so calculating that it uses everyone from opportunistic Russians, corrupt businessmen, an agoraphobic hacker--to a rock super star--to reach its goal. The Rock Star is the first thrilling book in the Rock Series trilogy.

Loss: MultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "gather_across_devices": false
}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: steps
per_device_train_batch_size: 64
per_device_eval_batch_size: 256
gradient_accumulation_steps: 4
learning_rate: 2e-05
warmup_ratio: 0.1
dataloader_num_workers: 2
remove_unused_columns: False
prompts: {'anchor_text': '', 'positive_text': ''}
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: steps
prediction_loss_only: True
per_device_train_batch_size: 64
per_device_eval_batch_size: 256
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 4
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 2e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 3.0
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_ratio: 0.1
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
bf16: False
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 2
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: False
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch_fused
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
project: huggingface
trackio_space_id: trackio
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
hub_revision: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_tokens_per_second: False
include_num_input_tokens_seen: no
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
liger_kernel_config: None
eval_use_gather_object: False
average_tokens_across_devices: True
prompts: {'anchor_text': '', 'positive_text': ''}
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss
0.1282	10	2.9737
0.2564	20	2.6733
0.3846	30	2.3228
0.5128	40	2.1395
0.6410	50	2.0539
0.7692	60	1.9516
0.8974	70	1.917
1.0256	80	1.9625
1.1538	90	1.8804
1.2821	100	1.8654
1.4103	110	1.8209
1.5385	120	1.8294
1.6667	130	1.8817
1.7949	140	1.9176
1.9231	150	1.9241
2.0513	160	1.9469
2.1795	170	1.8467
2.3077	180	1.8364
2.4359	190	1.8705
2.5641	200	1.8142
2.6923	210	1.8757
2.8205	220	1.8214
2.9487	230	1.8332

Framework Versions

Python: 3.12.12
Sentence Transformers: 5.2.2
Transformers: 4.57.6
PyTorch: 2.10.0+cu128
Accelerate: 1.12.0
Datasets: 4.3.0
Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}