Update README.md
Browse files
README.md
CHANGED
|
@@ -37,7 +37,7 @@ print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))
|
|
| 37 |
For **infill** sampling, we introduce three new special token types:
|
| 38 |
|
| 39 |
* `<mask_N>`: N-th span to be masked. In practice, use `<mask_1>` to where you want to sample infill.
|
| 40 |
-
* `<sep>`:
|
| 41 |
* `<eom>`: "End-Of-Mask" token that model will output at the end of infilling. You may use this token to truncate the output.
|
| 42 |
|
| 43 |
For example, if we want to generate infill for the following cursor position of a function:
|
|
|
|
| 37 |
For **infill** sampling, we introduce three new special token types:
|
| 38 |
|
| 39 |
* `<mask_N>`: N-th span to be masked. In practice, use `<mask_1>` to where you want to sample infill.
|
| 40 |
+
* `<sep>`: Separator token between the suffix and the infilled sample. See below.
|
| 41 |
* `<eom>`: "End-Of-Mask" token that model will output at the end of infilling. You may use this token to truncate the output.
|
| 42 |
|
| 43 |
For example, if we want to generate infill for the following cursor position of a function:
|