SmolLM3-3B-IT Fine-Tuned for Typst (Experimental)
An experimental QLoRA fine-tuned model for Typst code generation - shared for research and educational purposes
Warning: This is an experimental model that did not achieve production-quality results. It is shared for research transparency and educational purposes only.
Table of Contents
- SmolLM3-3B-IT Fine-Tuned for Typst (Experimental)
Model Summary
This model is a QLoRA fine-tuned version of HuggingFaceTB/SmolLM3-3B trained on the jalasoft/typst-instruct dataset for generating Typst markup language documents from natural language instructions.
| Attribute | Value |
|---|---|
| Base Model | HuggingFaceTB/SmolLM3-3B |
| Fine-tuning Method | QLoRA (4-bit quantization) |
| Training Framework | Axolotl |
| Dataset | jalasoft/typst-instruct (1,016 examples) |
| Task | Text-to-Typst code generation |
| Status | Experimental (not production-ready) |
Intended Use Cases (Not Achieved)
The model was developed with the goal of:
- Document generation: Creating complete Typst documents from natural language descriptions
- Template creation: Generating reusable Typst templates for various document types
However, the model did not meet the quality threshold required for these use cases due to the limitations described below.
Training Details
Training Procedure
- Method: QLoRA (Quantized Low-Rank Adaptation)
- Framework: Axolotl
- Precision: BF16 with 4-bit quantization
- Flash Attention: Enabled
Training Hyperparameters
| Parameter | Value |
|---|---|
| LoRA Rank (r) | 32 |
| LoRA Alpha | 96 |
| LoRA Dropout | 0.05 |
| LoRA Target Modules | q_proj, k_proj, v_proj, o_proj |
| Learning Rate | 2e-4 |
| LR Scheduler | Cosine |
| Optimizer | AdamW (fused) |
| Epochs | 3 |
| Sequence Length | 8192 |
| Micro Batch Size | 2 |
| Gradient Accumulation | 4 |
| Warmup Ratio | 0.1 |
| Weight Decay | 0.0 |
| Sample Packing | Enabled |
| Gradient Checkpointing | Enabled |
Training Data
- Dataset: jalasoft/typst-instruct
- Size: 1,016 instruction-completion pairs
- Content: Natural language instructions paired with Typst code and feature summaries
- Coverage: Academic documents, technical tutorials, reports, tables, figures, and advanced typography
Known Limitations
This model is marked as experimental due to significant issues observed during evaluation.
Observed Issues
| Issue | Description |
|---|---|
| Hallucinations | The model frequently generates incorrect or non-existent Typst syntax and features |
| Code Quality | Generated Typst code often contains syntax errors and fails to compile |
These limitations make the model unsuitable for production use in document generation workflows.
Recommendations
If you experiment with this model:
- Always validate generated Typst code before use
- Expect compilation errors in most complex task outputs
- Use for research purposes to understand fine-tuning challenges
- Consider larger models or more training data for production use cases
Evaluation
Performance was evaluated qualitatively across multiple model versions (v1, v2, v3) using 10 diverse Typst generation test cases. Results consistently fell below the quality threshold required for the intended use cases.
How to Use
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "jalasoft/smollm3-3B-it-ft-typ"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="float16")
messages = [
{
"role": "system",
"content": "You are an expert in Typst markup language. Generate clean, well-formatted Typst code based on user instructions:"
},
{
"role": "user",
"content": "Create a simple document with a title and two paragraphs"
}
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt")
outputs = model.generate(inputs, max_new_tokens=1000)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Citation and Licensing
License
This model is released under the MIT License. You are free to use, modify, and distribute this model for any purpose, including commercial applications.
Citation
If you use this model in your research, please cite:
@misc{smollm3-typst-experimental,
author = {{Jala R\&D}},
title = {SmolLM3-3B Fine-Tuned for Typst (Experimental)},
year = {2025},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/jalasoft/smollm3-3B-it-ft-typ}}
}
Additional Information
Model Card Authors
Created by Jalasoft R&D (@jalasoft)
Related Resources
- Training Dataset: jalasoft/typst-instruct
- Base Model: HuggingFaceTB/SmolLM3-3B
- Typst Official Documentation: https://typst.app/docs
- Typst GitHub Repository: https://github.com/typst/typst
- Downloads last month
- 2
Model tree for jalasoft/smollm3-3B-it-ft-typ
Base model
HuggingFaceTB/SmolLM3-3B-Base