Upload all models and assets for bjn (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.777
+    value: 4.828
   - name: best_isotropy
     type: isotropy
-    value: 0.8757
+    value: 0.8673
   - name: vocabulary_size
     type: vocab
-    value: 43482
-generated: 2025-12-28
+    value: 0
+generated: 2026-01-03
 ---
 
 # BJN - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,51 +70,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.698x | 3.62 | 0.3416% | 403,663 |
-| **16k** | 4.081x | 4.00 | 0.3770% | 365,824 |
-| **32k** | 4.450x | 4.36 | 0.4111% | 335,441 |
-| **64k** | 4.777x 🏆 | 4.68 | 0.4413% | 312,459 |
+| **8k** | 3.761x | 3.76 | 0.3901% | 369,664 |
+| **16k** | 4.163x | 4.17 | 0.4318% | 333,925 |
+| **32k** | 4.537x | 4.54 | 0.4706% | 306,394 |
+| **64k** | 4.828x 🏆 | 4.83 | 0.5008% | 287,926 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Makmur yaitu sabuting kampung di Kacamatan Angsana, Kabupatin Tanah Bumbu, Prupi...`
+**Sample 1:** `Gunung Anyar adalah sabuting kacamatan di Kuta Surabaya, Prupinsi Jawa Timur, In...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁makmur ▁yaitu ▁sabuting ▁kampung ▁di ▁kacamatan ▁ angs ana , ... (+10 more)` | 20 |
-| 16k | `▁makmur ▁yaitu ▁sabuting ▁kampung ▁di ▁kacamatan ▁angsana , ▁kabupatin ▁tanah ... (+8 more)` | 18 |
-| 32k | `▁makmur ▁yaitu ▁sabuting ▁kampung ▁di ▁kacamatan ▁angsana , ▁kabupatin ▁tanah ... (+8 more)` | 18 |
-| 64k | `▁makmur ▁yaitu ▁sabuting ▁kampung ▁di ▁kacamatan ▁angsana , ▁kabupatin ▁tanah ... (+8 more)` | 18 |
+| 8k | `▁gunung ▁anyar ▁adalah ▁sabuting ▁kacamatan ▁di ▁kuta ▁surabaya , ▁prupinsi ... (+6 more)` | 16 |
+| 16k | `▁gunung ▁anyar ▁adalah ▁sabuting ▁kacamatan ▁di ▁kuta ▁surabaya , ▁prupinsi ... (+6 more)` | 16 |
+| 32k | `▁gunung ▁anyar ▁adalah ▁sabuting ▁kacamatan ▁di ▁kuta ▁surabaya , ▁prupinsi ... (+6 more)` | 16 |
+| 64k | `▁gunung ▁anyar ▁adalah ▁sabuting ▁kacamatan ▁di ▁kuta ▁surabaya , ▁prupinsi ... (+6 more)` | 16 |
 
-**Sample 2:** `Tanjung Mangkalihat adalah sabuah kampung di Kacamatan Sandaran, Kabupatin Kutai...`
+**Sample 2:** `Tebedak adalah sabuah kampung di Kacamatan Ngabang, Kabupatin Landak, Prupinsi K...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁tanjung ▁mangk alihat ▁adalah ▁sabuah ▁kampung ▁di ▁kacamatan ▁sand aran ... (+11 more)` | 21 |
-| 16k | `▁tanjung ▁mangk alihat ▁adalah ▁sabuah ▁kampung ▁di ▁kacamatan ▁sandaran , ... (+10 more)` | 20 |
-| 32k | `▁tanjung ▁mangk alihat ▁adalah ▁sabuah ▁kampung ▁di ▁kacamatan ▁sandaran , ... (+10 more)` | 20 |
-| 64k | `▁tanjung ▁mangk alihat ▁adalah ▁sabuah ▁kampung ▁di ▁kacamatan ▁sandaran , ... (+10 more)` | 20 |
+| 8k | `▁teb ed ak ▁adalah ▁sabuah ▁kampung ▁di ▁kacamatan ▁ngabang , ... (+9 more)` | 19 |
+| 16k | `▁teb edak ▁adalah ▁sabuah ▁kampung ▁di ▁kacamatan ▁ngabang , ▁kabupatin ... (+8 more)` | 18 |
+| 32k | `▁teb edak ▁adalah ▁sabuah ▁kampung ▁di ▁kacamatan ▁ngabang , ▁kabupatin ... (+8 more)` | 18 |
+| 64k | `▁tebedak ▁adalah ▁sabuah ▁kampung ▁di ▁kacamatan ▁ngabang , ▁kabupatin ▁landak ... (+7 more)` | 17 |
 
-**Sample 3:** `Sumuragung adalah sabuah kampung di Kacamatan Sumberejo, Kabupatin Bojonegoro, P...`
+**Sample 3:** `Handil Birayang Atas yaitu sabuting kampung di Kacamatan Bumi Makmur, Kabupatin ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁sum ur agung ▁adalah ▁sabuah ▁kampung ▁di ▁kacamatan ▁sumberejo , ... (+9 more)` | 19 |
-| 16k | `▁sumur agung ▁adalah ▁sabuah ▁kampung ▁di ▁kacamatan ▁sumberejo , ▁kabupatin ... (+8 more)` | 18 |
-| 32k | `▁sumur agung ▁adalah ▁sabuah ▁kampung ▁di ▁kacamatan ▁sumberejo , ▁kabupatin ... (+8 more)` | 18 |
-| 64k | `▁sumur agung ▁adalah ▁sabuah ▁kampung ▁di ▁kacamatan ▁sumberejo , ▁kabupatin ... (+8 more)` | 18 |
+| 8k | `▁handil ▁bir ayang ▁atas ▁yaitu ▁sabuting ▁kampung ▁di ▁kacamatan ▁bumi ... (+12 more)` | 22 |
+| 16k | `▁handil ▁bir ayang ▁atas ▁yaitu ▁sabuting ▁kampung ▁di ▁kacamatan ▁bumi ... (+12 more)` | 22 |
+| 32k | `▁handil ▁birayang ▁atas ▁yaitu ▁sabuting ▁kampung ▁di ▁kacamatan ▁bumi ▁makmur ... (+11 more)` | 21 |
+| 64k | `▁handil ▁birayang ▁atas ▁yaitu ▁sabuting ▁kampung ▁di ▁kacamatan ▁bumi ▁makmur ... (+11 more)` | 21 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 4.777x compression
-- **Lowest UNK Rate:** 8k with 0.3416% unknown tokens
+- **Best Compression:** 64k achieves 4.828x compression
+- **Lowest UNK Rate:** 8k with 0.3901% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -121,57 +129,89 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 8,214 🏆 | 13.00 | 31,697 | 22.3% | 43.3% |
-| **2-gram** | 216 🏆 | 7.75 | 3,606 | 75.0% | 99.2% |
-| **3-gram** | 7,363 | 12.85 | 33,190 | 25.9% | 45.0% |
-| **3-gram** | 1,713 | 10.74 | 25,998 | 32.5% | 77.1% |
-| **4-gram** | 9,532 | 13.22 | 46,645 | 25.7% | 43.0% |
-| **4-gram** | 9,186 | 13.17 | 125,660 | 16.5% | 47.6% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 6,379 | 12.64 | 21,446 | 23.8% | 44.7% |
+| **2-gram** | Subword | 186 🏆 | 7.54 | 2,773 | 78.2% | 99.5% |
+| **3-gram** | Word | 3,749 | 11.87 | 17,540 | 32.7% | 52.0% |
+| **3-gram** | Subword | 1,430 | 10.48 | 20,264 | 34.4% | 80.3% |
+| **4-gram** | Word | 5,222 | 12.35 | 24,563 | 29.0% | 48.3% |
+| **4-gram** | Subword | 7,617 | 12.90 | 99,301 | 17.6% | 50.0% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `kampung di` | 5,958 |
+| 2 | `prupinsi kalimantan` | 5,878 |
+| 3 | `di kacamatan` | 5,619 |
+| 4 | `adalah sabuah` | 4,219 |
+| 5 | `sabuah kampung` | 3,812 |
+
+**3-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `kampung di kacamatan` | 5,212 |
+| 2 | `adalah sabuah kampung` | 3,811 |
+| 3 | `sabuah kampung di` | 3,810 |
+| 4 | `kalimantan selatan indunisia` | 2,206 |
+| 5 | `prupinsi kalimantan selatan` | 2,192 |
+
+**4-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `sabuah kampung di kacamatan` | 3,810 |
+| 2 | `adalah sabuah kampung di` | 3,809 |
+| 3 | `prupinsi kalimantan selatan indunisia` | 2,158 |
+| 4 | `prupinsi kalimantan barat indunisia` | 1,806 |
+| 5 | `sabuting kampung di kacamatan` | 1,342 |
+
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `indunisia .` | 7,623 |
-| 2 | `, indunisia` | 7,382 |
-| 3 | `, prupinsi` | 7,240 |
-| 4 | `, kabupatin` | 6,196 |
-| 5 | `kampung di` | 5,960 |
+| 1 | `a n` | 360,335 |
+| 2 | `n _` | 192,183 |
+| 3 | `n g` | 150,787 |
+| 4 | `a _` | 137,197 |
+| 5 | `k a` | 130,918 |
 
-**3-grams:**
+**3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `, indunisia .` | 7,294 |
-| 2 | `, prupinsi kalimantan` | 5,775 |
-| 3 | `kampung di kacamatan` | 5,212 |
-| 4 | `sabuah kampung di` | 3,803 |
-| 5 | `adalah sabuah kampung` | 3,803 |
+| 1 | `a n _` | 154,048 |
+| 2 | `a n g` | 83,639 |
+| 3 | `_ k a` | 75,768 |
+| 4 | `n g _` | 74,596 |
+| 5 | `_ m a` | 57,180 |
 
-**4-grams:**
+**4-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `sabuah kampung di kacamatan` | 3,802 |
-| 2 | `adalah sabuah kampung di` | 3,801 |
-| 3 | `kalimantan selatan , indunisia` | 2,200 |
-| 4 | `selatan , indunisia .` | 2,195 |
-| 5 | `prupinsi kalimantan selatan ,` | 2,161 |
+| 1 | `a n g _` | 48,221 |
+| 2 | `t a n _` | 34,250 |
+| 3 | `n a n g` | 29,540 |
+| 4 | `a t a n` | 29,135 |
+| 5 | `_ n a n` | 28,221 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 216
+- **Best Perplexity:** 2-gram (subword) with 186
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~48% of corpus
+- **Coverage:** Top-1000 patterns cover ~50% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -179,55 +219,86 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.7609 | 1.695 | 5.41 | 108,020 | 23.9% |
-| **1** | 0.9104 | 1.880 | 5.41 | 2,572 | 9.0% |
-| **2** | 0.2888 | 1.222 | 1.67 | 583,707 | 71.1% |
-| **2** | 0.7263 | 1.654 | 4.39 | 13,908 | 27.4% |
-| **3** | 0.0836 | 1.060 | 1.14 | 970,699 | 91.6% |
-| **3** | 0.7940 | 1.734 | 3.86 | 61,013 | 20.6% |
-| **4** | 0.0265 🏆 | 1.019 | 1.04 | 1,101,630 | 97.4% |
-| **4** | 0.6245 🏆 | 1.542 | 2.66 | 235,356 | 37.6% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.8466 | 1.798 | 5.65 | 98,412 | 15.3% |
+| **1** | Subword | 0.7154 | 1.642 | 4.59 | 2,413 | 28.5% |
+| **2** | Word | 0.2332 | 1.175 | 1.48 | 553,823 | 76.7% |
+| **2** | Subword | 0.6822 | 1.605 | 4.16 | 11,071 | 31.8% |
+| **3** | Word | 0.0559 | 1.040 | 1.08 | 814,594 | 94.4% |
+| **3** | Subword | 0.7802 | 1.717 | 3.89 | 46,029 | 22.0% |
+| **4** | Word | 0.0149 🏆 | 1.010 | 1.02 | 878,428 | 98.5% |
+| **4** | Subword | 0.6536 | 1.573 | 2.81 | 179,116 | 34.6% |
+
+### Generated Text Samples (Word-based)
+
+Below are text samples generated from each word-based Markov chain model:
 
-### Generated Text Samples
+**Context Size 1:**
+
+1. `di sarawak ihwal nitu pamimpin national league soccer wan upananda diikat albumin salain satelitnya ...`
+2. `nang kaya talabang pinggir papila yaitu hutan hujan pada sidang agung nang disuruh rayi rayiading ad...`
+3. `wan teolog kadada dana saganal ganalnya ujak langsung maka kata lawan kakawanannya sesama youtuber p...`
+
+**Context Size 2:**
+
+1. `kampung di wilayah kacamatan sei menggaris pambagian administratip kacamatan batu ampar pambagian ad...`
+2. `prupinsi kalimantan barat indunisia géografi watas wilayah kampung baru yaitu sabuting kampung di ka...`
+3. `di kacamatan gambut kabupatin banjar prupinsi kalimantan timur indunisia géografi watas wilayah wata...`
+
+**Context Size 3:**
+
+1. `kampung di kacamatan babat kabupatin lamongan prupinsi jawa timur indunisia jujuhutan`
+2. `adalah sabuah kampung di kacamatan parindu kabupatin sanggau prupinsi kalimantan barat indunisia géo...`
+3. `sabuah kampung di kacamatan solokuro kabupatin lamongan prupinsi jawa timur indunisia jujuhutan`
+
+**Context Size 4:**
 
-Below are text samples generated from each Markov chain model:
+1. `sabuah kampung di kacamatan muara badak kabupatin kutai kartanegara prupinsi kalimantan timur induni...`
+2. `adalah sabuah kampung di kacamatan kota bangun kabupatin kutai kartanegara prupinsi kalimantan timur...`
+3. `sabuting kampung di kacamatan pulau laut timur kabupatin kotabaru prupinsi kalimantan selatan induni...`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `, umat buddha dingarani nangkaya : kampung di kacamatan pengaron sambung makmur adalah pangalola wan...`
-2. `. jabatan dalam bintang nang malili di kacamatan tulin onsoi ngaran rasmi ; 3 . elly`
-3. `di antara ras , wan bawah tanah siang sampai tahun 1978 wan knapstad , antara 28`
+1. `ah_beratelancit_`
+2. `_k_harangipin_pa`
+3. `n_seajudan_ancar`
 
 **Context Size 2:**
 
-1. `, indunisia . géografi watas wilayah kacamatan telaga antang ngaran kacamatan di kabupatin bengkayan...`
-2. `, prupinsi kalimantan barat , indunisia . géografi watas wilayah kampung mawar sari yaitu sabuting k...`
-3. `, kabupatin bangkalan , prupinsi kalimantan selatan , indunisia . géografi watas wilayah kampung bin...`
+1. `antar_hunimungara`
+2. `n_vilet,_dimbuan_`
+3. `ng._ikuah_nanti_h`
 
 **Context Size 3:**
 
-1. `, prupinsi kalimantan selatan , indunisia . watas wilayah watas wilayah kacamatan sepauk : pambagian...`
-2. `kampung di kacamatan selakau , kabupatin sambas , prupinsi kalimantan selatan , indunisia . watas wi...`
-3. `adalah sabuah kampung di kacamatan sungai tabuk , kabupatin banjar , prupinsi kalimantan barat , ind...`
+1. `an_hijau_duwa_dino`
+2. `angai_wataceh_tan_`
+3. `_kamiri'iai_(di_ka`
 
 **Context Size 4:**
 
-1. `sabuah kampung di kacamatan tabang , kabupatin kutai kartanegara , prupinsi kalimantan timur , indun...`
-2. `adalah sabuah kampung di kacamatan glagah , kabupatin lamongan , prupinsi jawa timur , indunisia . g...`
-3. `kalimantan selatan , indunisia . géografi watas wilayah watas wilayah kacamatan gunungpuyuh : pambag...`
+1. `ang_paningkayat_nan`
+2. `tan_sabalumnya,_jum`
+3. `nang_labu_adalah_ra`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 97.4% predictability
+- **Best Predictability:** Context-4 (word) with 98.5% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (235,356 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (179,116 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -243,64 +314,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 43,482 |
-| Total Tokens | 1,036,384 |
-| Mean Frequency | 23.83 |
+| Vocabulary Size | 41,097 |
+| Total Tokens | 980,207 |
+| Mean Frequency | 23.85 |
 | Median Frequency | 4 |
-| Frequency Std Dev | 273.79 |
+| Frequency Std Dev | 276.25 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | di | 27,734 |
-| 2 | nang | 27,434 |
-| 3 | wan | 17,268 |
-| 4 | adalah | 10,732 |
-| 5 | lawan | 9,615 |
-| 6 | indunisia | 9,432 |
-| 7 | kacamatan | 9,151 |
-| 8 | kalimantan | 8,405 |
-| 9 | kampung | 7,829 |
-| 10 | matan | 7,719 |
+| 1 | di | 27,381 |
+| 2 | nang | 26,981 |
+| 3 | wan | 16,949 |
+| 4 | adalah | 10,740 |
+| 5 | indunisia | 9,397 |
+| 6 | lawan | 9,395 |
+| 7 | kacamatan | 9,110 |
+| 8 | kalimantan | 8,310 |
+| 9 | kampung | 7,769 |
+| 10 | matan | 7,559 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | bepacaran | 2 |
-| 2 | beregszásziová | 2 |
-| 3 | košice | 2 |
-| 4 | satian | 2 |
-| 5 | frisna | 2 |
-| 6 | ropang | 2 |
-| 7 | caknan | 2 |
-| 8 | muktamar | 2 |
-| 9 | sandon | 2 |
-| 10 | sékuéns | 2 |
+| 1 | subreddit | 2 |
+| 2 | falco | 2 |
+| 3 | altarnatif | 2 |
+| 4 | elevasi | 2 |
+| 5 | kapus | 2 |
+| 6 | klayangan | 2 |
+| 7 | simbangan | 2 |
+| 8 | basyah | 2 |
+| 9 | simanggu | 2 |
+| 10 | kaliningan | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.0456 |
-| R² (Goodness of Fit) | 0.994175 |
+| Zipf Coefficient | 1.0489 |
+| R² (Goodness of Fit) | 0.995015 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 34.5% |
-| Top 1,000 | 61.2% |
-| Top 5,000 | 80.9% |
-| Top 10,000 | 88.2% |
+| Top 100 | 35.5% |
+| Top 1,000 | 62.3% |
+| Top 5,000 | 81.6% |
+| Top 10,000 | 88.8% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9942 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 34.5% of corpus
-- **Long Tail:** 33,482 words needed for remaining 11.8% coverage
+- **Zipf Compliance:** R²=0.9950 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 35.5% of corpus
+- **Long Tail:** 31,097 words needed for remaining 11.2% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -313,24 +384,134 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 21,102 | 32 | 3.645 | 0.820 | 0.8757 🏆 |
-| **mono_64d** | 21,102 | 64 | 4.192 | 0.774 | 0.8364 |
-| **mono_128d** | 21,102 | 128 | 4.539 | 0.732 | 0.5631 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.8673 🏆 | 0.3425 | N/A | N/A |
+| **mono_64d** | 64 | 0.8381 | 0.2559 | N/A | N/A |
+| **mono_128d** | 128 | 0.5573 | 0.2314 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.8757 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 21,102 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.8673 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.2766. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-ma` | malayisasi, maajuakan, mamake |
+| `-pa` | padatuannya, parlumbaan, panyalidikan |
+| `-di` | dirasmikan, ditarima, diversity |
+| `-ba` | babanyu, bacalak, badatang |
+| `-ka` | karaktir, kavi, kailmuan |
+| `-me` | melimpah, mesoamerika, menceritakan |
+| `-man` | manuntun, manguap, mangurbanakan |
+| `-pe` | perhubungan, perpaduan, peraih |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-n` | perhubungan, dirasmikan, parlumbaan |
+| `-an` | perhubungan, dirasmikan, parlumbaan |
+| `-a` | padatuannya, ditarima, arménia |
+| `-ng` | sekapung, jelutung, badatang |
+| `-ya` | padatuannya, syairnya, sautingnya |
+| `-nya` | padatuannya, syairnya, sautingnya |
+| `-kan` | dirasmikan, maajuakan, panyalidikan |
+| `-akan` | maajuakan, menceritakan, mambedakan |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `anga` | 1.72x | 223 contexts | tanga, sanga, angah |
+| `unga` | 1.95x | 57 contexts | bunga, tungau, sungay |
+| `ntan` | 1.94x | 48 contexts | antan, intan, rentan |
+| `ngan` | 1.85x | 58 contexts | bongan, ringan, mangan |
+| `anja` | 1.68x | 82 contexts | ganja, anjat, sanja |
+| `ting` | 1.59x | 79 contexts | piting, tingah, eating |
+| `ndun` | 2.11x | 23 contexts | rundun, indung, bendung |
+| `mant` | 1.80x | 39 contexts | manta, manti, mantah |
+| `dala` | 1.69x | 38 contexts | dalas, dalam, adalah |
+| `pung` | 1.88x | 21 contexts | apung, tapung, oppung |
+| `atin` | 1.75x | 26 contexts | patin, satin, atina |
+| `mpun` | 1.61x | 27 contexts | ampun, impun, rumpun |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-pa` | `-n` | 239 words | panulisan, panjagaan |
+| `-pa` | `-an` | 228 words | panulisan, panjagaan |
+| `-di` | `-n` | 165 words | dibandingakan, didasarakan |
+| `-di` | `-an` | 159 words | dibandingakan, didasarakan |
+| `-ma` | `-n` | 156 words | manurunakan, manyaurangan |
+| `-di` | `-kan` | 155 words | dibandingakan, didasarakan |
+| `-ma` | `-an` | 145 words | manurunakan, manyaurangan |
+| `-ka` | `-n` | 144 words | kaharuddin, kawarganegaraan |
+| `-ka` | `-an` | 138 words | kawarganegaraan, katabalan |
+| `-ma` | `-kan` | 127 words | manurunakan, mahamburakan |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| kadatangannya | **`ka-data-ng-an-nya`** | 9.0 | `data` |
+| dikaluarkannya | **`di-ka-luar-kan-nya`** | 9.0 | `luar` |
+| badatangan | **`ba-data-ng-an`** | 7.5 | `data` |
+| dibayangakan | **`di-ba-yang-akan`** | 7.5 | `yang` |
+| kahabangan | **`ka-haba-ng-an`** | 7.5 | `haba` |
+| ditinggalakannya | **`di-tinggal-akan-nya`** | 7.5 | `tinggal` |
+| kakuasaannya | **`ka-kuasa-an-nya`** | 7.5 | `kuasa` |
+| didinginakan | **`di-di-ngin-akan`** | 7.5 | `ngin` |
+| mamakainya | **`ma-ma-ka-inya`** | 7.5 | `inya` |
+| kaakhirannya | **`ka-akhir-an-nya`** | 7.5 | `akhir` |
+| kadalamannya | **`ka-dalam-an-nya`** | 7.5 | `dalam` |
+| katakutanan | **`ka-takut-an-an`** | 7.5 | `takut` |
+| kanyataannya | **`ka-nyata-an-nya`** | 7.5 | `nyata` |
+| dikeluarakan | **`di-keluar-akan`** | 6.0 | `keluar` |
+| batanaman | **`ba-tanam-an`** | 6.0 | `tanam` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language BJN appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -338,11 +519,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.78x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (216) |
-| Markov | **Context-4** | Highest predictability (97.4%) |
+| Tokenizer | **64k BPE** | Best compression (4.83x) |
+| N-gram | **2-gram** | Lowest perplexity (186) |
+| Markov | **Context-4** | Highest predictability (98.5%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -532,7 +714,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -548,7 +731,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-28 05:18:09*
+*Report Date: 2026-01-03 07:24:26*

models/embeddings/monolingual/bjn_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e65409ff41e4182c5ab1696db0a4dd291168ff75e7a3d84800d52d65677827b8
-size 1045964232
+oid sha256:e4e093c6a31e77a391c65fb05175223839736144a137894bb279acedb38a739e
+size 1044759476

models/embeddings/monolingual/bjn_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 21102
+  "vocab_size": 19945
 }

models/embeddings/monolingual/bjn_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6c4ba0d5d7cc97005bb29d178bcc33da604ec2ff54b23f1f552000768cabdbf8
-size 261757896
+oid sha256:5dc1832ebecd5f5dafd473e3651dd39adeeb3b7babccfe0f41a0ff07bead9833
+size 261441716

models/embeddings/monolingual/bjn_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 21102
+  "vocab_size": 19945
 }

models/embeddings/monolingual/bjn_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:200944aec2429d1617154426e2e8a0393267c3a2b5366e032a99e04000daf227
-size 523160008
+oid sha256:01dad40fd9ad207e7764f6c7896ea76ca31b58aac8fc24673d9cc7aaf6e7e1dd
+size 522547636

models/embeddings/monolingual/bjn_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 21102
+  "vocab_size": 19945
 }

models/subword_markov/bjn_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4e9986331e278cde13c4581c7a4b37a46d7a9c5a6972ad0119dec3127a4a9468
-size 105165
+oid sha256:7ada7af32fa948a0655b7ebf55bc43135b6b1ac0a6182109dbe7b9ab245e5697
+size 91151

models/subword_markov/bjn_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "bjn",
-  "unique_contexts": 2572,
-  "total_transitions": 7876088
+  "unique_contexts": 2413,
+  "total_transitions": 7333167
 }

models/subword_markov/bjn_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:62714b996e21cfc6c73dffa6ddadff23b1cb4ebbb512ea7979b9607d639f9e52
-size 495942
+oid sha256:ad84eb896f07adf50c1c8330a4fd412521e79c8611f16dbd54db03976d031b2a
+size 394385

models/subword_markov/bjn_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "bjn",
-  "unique_contexts": 13908,
-  "total_transitions": 7864568
+  "unique_contexts": 11071,
+  "total_transitions": 7321716
 }

models/subword_markov/bjn_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:21487466bfac2e926216be93e702aaf0111e80998a59a47c1d5f6d60bb3e55ae
-size 1728073
+oid sha256:629f8e2533e8eb22ae3400a41bd35f7de6abee27c253d69864fba53bf91a7cab
+size 1413172

models/subword_markov/bjn_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "bjn",
-  "unique_contexts": 61013,
-  "total_transitions": 7853048
+  "unique_contexts": 46029,
+  "total_transitions": 7310265
 }

models/subword_markov/bjn_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:40577f595c4773cf05fd06e7676f078a7def769251fd22df537a94333ef0d960
-size 4742991
+oid sha256:1f91a26206e37ee242d2c77719cfea85fb5572aabe4780ad4ddc16f7661bef71
+size 3779369

models/subword_markov/bjn_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "bjn",
-  "unique_contexts": 235356,
-  "total_transitions": 7841528
+  "unique_contexts": 179116,
+  "total_transitions": 7298814
 }

models/subword_ngram/bjn_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:947ebba2eea6709955c942ae6a45f98916212ba270e6d663b77823d2c8442b1d
-size 46094
+oid sha256:6f14b3c618cdbe3ba5b783d54065bf7f34a9d7f1b4144d0953086f77baae8376
+size 37041

models/subword_ngram/bjn_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "bjn",
-  "unique_ngrams": 3606,
-  "total_ngrams": 7876088
+  "unique_ngrams": 2773,
+  "total_ngrams": 7333167
 }

models/subword_ngram/bjn_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3d3d72e0319456f87568ec49b946d626c3ff9f845502be1f8d3d69d8c4dac8db
-size 311919
+oid sha256:353eff0669e301d0a373b59a8360bbc5dca472bf6870e145137cb0b3ea1388df
+size 245202

models/subword_ngram/bjn_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "bjn",
-  "unique_ngrams": 25998,
-  "total_ngrams": 7864568
+  "unique_ngrams": 20264,
+  "total_ngrams": 7321716
 }

models/subword_ngram/bjn_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:78caf0f27c30dc7aaf8637610d8d16e7e8bf3bb39788eb3d95c3ad4faae93613
-size 1385663
+oid sha256:a4f49217b5541a90584ec8de80222bf07caa7874767428ec91809828776b503d
+size 1113155

models/subword_ngram/bjn_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "bjn",
-  "unique_ngrams": 125660,
-  "total_ngrams": 7853048
+  "unique_ngrams": 99301,
+  "total_ngrams": 7310265
 }

models/tokenizer/bjn_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0789678d80af8c0732774eb4b6bb1e41146fd09072f82089a4bb2aa3a90517c8
-size 505456
+oid sha256:c3f5b630291acf2ba47ddad779b9b7da931da16ba83757be5051cc96de210969
+size 506686

models/tokenizer/bjn_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4919663d1e73201b9de368a024ef7e7fb2b833b887eb7c39a545e5b90b0f1b6d
-size 780920
+oid sha256:909287240aaef4cf222642164f43a915d96db473274547f67f0fad9b37c325b0
+size 783745

models/tokenizer/bjn_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:84987346017d033ed1a7339bbb8ad76c2fc1c9f7824e8d1912cfa72d31384a80
-size 1331984
+oid sha256:b486f1ebd0107b543a8cc67ef5b38a3e84bc04e29bc7c4a065808b100089559d
+size 1358958

models/tokenizer/bjn_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f08a90f9e0b51994122ed4bb0a18d36e598dfea67df86f18ae6cc765079db22e
-size 371550
+oid sha256:12bb0d65ad2471af55c560a13bb81daac89ac54df022206918fc85d8bd8cbf7a
+size 372387

models/vocabulary/bjn_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:176907dc0990fd1f1a9773b5c9d088fb3e3e76c41eb4e4335673c1e3550bc153
-size 741453
+oid sha256:89e712e174d699814d715ee756db596976724acb27e04ba9b879b084c92b678b
+size 697891

models/vocabulary/bjn_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "bjn",
-  "vocabulary_size": 43482,
+  "vocabulary_size": 41097,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.09796208341358296,
+    "type_token_ratio": 0.09502040061134379,
     "coverage": {
-      "top_100": 0.32512364499260493,
-      "top_1000": 0.576528164893907,
-      "top_5000": 0.761968399210707,
-      "top_10000": 0.8305046096597537
+      "top_100": 0.3351626748795911,
+      "top_1000": 0.5889079264614335,
+      "top_5000": 0.7705822606228692,
+      "top_10000": 0.8384362165297953
     },
-    "hapax_count": 64348,
-    "hapax_ratio": 0.5967541500510062,
-    "total_documents": 11520
+    "hapax_count": 57507,
+    "hapax_ratio": 0.5832116344164537,
+    "total_documents": 11451
   }
 }

models/word_markov/bjn_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0e1e490ec45c03263bbee5b0cb5d14a9309b3856b41d89553247da0fad241778
-size 4355355
+oid sha256:be08d60f4e38caedbe81491c069935efc94c9f044fae86f70e1ce4a81d7d2269
+size 3894998

models/word_markov/bjn_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "bjn",
-  "unique_contexts": 108020,
-  "total_transitions": 1341605
+  "unique_contexts": 98412,
+  "total_transitions": 1026263
 }

models/word_markov/bjn_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:31270f3260dd0112d37d7fc2f7d4231f5a1ee74904fee4b7d4ed309f3add4d7f
-size 11014789
+oid sha256:5b38f524f3d4a7fc742e5e51668a0ad8b6f77409c9a3d88cabe8baeca1b75ce5
+size 9782557

models/word_markov/bjn_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "bjn",
-  "unique_contexts": 583707,
-  "total_transitions": 1330085
+  "unique_contexts": 553823,
+  "total_transitions": 1014812
 }

models/word_markov/bjn_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6dbbda53445631ccecd1be92001ce4f423233d3fe041ffb12afe0a356c91a9ef
-size 16006739
+oid sha256:9e4641a1bb879c32589191b75a65c70e3190c7c7012dc07fb43464b661574095
+size 13384380

models/word_markov/bjn_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "bjn",
-  "unique_contexts": 970699,
-  "total_transitions": 1318565
+  "unique_contexts": 814594,
+  "total_transitions": 1003361
 }

models/word_markov/bjn_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c1bb5f64ee3e9ecd1c820d8a4acba751e1066a6106871f181251a4debf90ddba
-size 18389438
+oid sha256:589c81fdb080e15fee3973c1bf503f9793839faa5df1188dfe36ec5012423753
+size 15173143

models/word_markov/bjn_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "bjn",
-  "unique_contexts": 1101630,
-  "total_transitions": 1307048
+  "unique_contexts": 878428,
+  "total_transitions": 991910
 }

models/word_ngram/bjn_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cd6e87d662bfb47bba380d18db5125e72bf837e8638bf8005e5b38bea2e82aab
-size 438250
+oid sha256:6c296c4547051fe6c63b544b17d1841b9c38a3358271cb5be47cba467fadc44e
+size 310635

models/word_ngram/bjn_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "bjn",
-  "unique_ngrams": 31697,
-  "total_ngrams": 1341605
+  "unique_ngrams": 21446,
+  "total_ngrams": 1026263
 }

models/word_ngram/bjn_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ef08ff903e4ce470871df54fd8f6d35ed89c905fe3c1fcfd357809fa41aa7266
-size 523283
+oid sha256:30e0c8650bfeaa8700456c4ebfebab028f6bd3eff3cc95ca48702bcb4696195d
+size 284550

models/word_ngram/bjn_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "bjn",
-  "unique_ngrams": 33190,
-  "total_ngrams": 1330085
+  "unique_ngrams": 17540,
+  "total_ngrams": 1014812
 }

models/word_ngram/bjn_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3f632f15f1987aec2e7d1af2a418ac4cf9f9f426b023424d33b864115d7ae130
-size 732377
+oid sha256:f04fd3b0c91837b04ad16f7225dddfc0bd12d9d4cdf0ec005c1dbc42d921d9ba
+size 433479

models/word_ngram/bjn_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "bjn",
-  "unique_ngrams": 46645,
-  "total_ngrams": 1318565
+  "unique_ngrams": 24563,
+  "total_ngrams": 1003361
 }