tf-tpu
/

unigram-tokenizer-wikitext

Model card Files Files and versions

sayakpaul HF Staff commited on Feb 16, 2023

Commit

60380a1

·

1 Parent(s): 849180a

Upload tokenizer

Files changed (1) hide show

tokenizer.json +7 -7

tokenizer.json CHANGED Viewed

@@ -179,19 +179,19 @@
       ],
       [
         "t",
-        -0.9165908018741468
       ],
       [
-        "e",
-        -1.610238232433968
       ],
       [
-        "x",
-        -1.610238232433968
       ],
       [
-        "▁",
-        -1.610238232433968
       ]
     ]
   }

       ],
       [
         "t",
+        -0.9469899237864784
       ],
       [
+        "▁",
+        -1.6926248444729384
       ],
       [
+        "e",
+        -1.6926248444729384
       ],
       [
+        "x",
+        -1.6926248444729384
       ]
     ]
   }