Clarify pre-tokenize before multigpu (#359)
Browse files
README.md
CHANGED
|
@@ -524,7 +524,14 @@ Run
|
|
| 524 |
accelerate launch scripts/finetune.py configs/your_config.yml
|
| 525 |
```
|
| 526 |
|
| 527 |
-
#### Multi-GPU
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 528 |
|
| 529 |
- llama FSDP
|
| 530 |
```yaml
|
|
|
|
| 524 |
accelerate launch scripts/finetune.py configs/your_config.yml
|
| 525 |
```
|
| 526 |
|
| 527 |
+
#### Multi-GPU
|
| 528 |
+
|
| 529 |
+
It is recommended to pre-tokenize dataset with the following before finetuning:
|
| 530 |
+
```bash
|
| 531 |
+
CUDA_VISIBLE_DEVICES="" accelerate ... --prepare_ds_only
|
| 532 |
+
```
|
| 533 |
+
|
| 534 |
+
##### Config
|
| 535 |
|
| 536 |
- llama FSDP
|
| 537 |
```yaml
|