eliebak
/

the-tokenizer-v1.5

Model card Files Files and versions

eliebak HF Staff commited on Jun 27, 2024

Commit

80079d5

·

verified ·

1 Parent(s): e418e1e

Update README.md

Files changed (1) hide show

README.md +2 -2

README.md CHANGED Viewed

@@ -6,8 +6,8 @@ tags: []
 # Model Card for Model ID
 `lvwerra/the-tokenizer-v1` but replacing:
-- <fim_prefix> -> <|im_start|> (token id 1)
-- <fim_middle> -> <|im_end|> (token id 2)

 # Model Card for Model ID
 `lvwerra/the-tokenizer-v1` but replacing:
+- <\fim_prefix> -> <|im_start|> (token id 1)
+- <\fim_middle> -> <|im_end|> (token id 2)