Spaces:

Dovakiins
/

qwerrwe

Build error

Nanobit commited on Aug 12, 2023

Commit

289d5c4

unverified ·

1 Parent(s): 35c8b90

feat(merge): save tokenizer on merge (#362)

Files changed (1) hide show

scripts/finetune.py CHANGED Viewed

@@ -265,6 +265,7 @@ def train(
         if cfg.local_rank == 0:
             LOG.info("saving merged model")
             model.save_pretrained(str(Path(cfg.output_dir) / "merged"))
         return
     if cfg.inference:

         if cfg.local_rank == 0:
             LOG.info("saving merged model")
             model.save_pretrained(str(Path(cfg.output_dir) / "merged"))
+            tokenizer.save_pretrained(str(Path(cfg.output_dir) / "merged"))
         return
     if cfg.inference: