Achal Dave
commited on
Commit
·
2fbae44
1
Parent(s):
62f139c
Update
Browse files
README.md
CHANGED
|
@@ -13,13 +13,13 @@ DCLM-1B is a 1.4 billion parameter language model trained on the DCLM-Baseline d
|
|
| 13 |
|
| 14 |
## Evaluation
|
| 15 |
|
| 16 |
-
|
| 17 |
As described in the paper, Core accuracy is the average of centered accuracy on
|
| 18 |
22 tasks (including HellaSwag and ARC-E), Extended is centered accuracy averaged
|
| 19 |
over 53 tasks.
|
| 20 |
|
| 21 |
|
| 22 |
-
| Model | Params | Tokens | Open dataset? | Core | MMLU
|
| 23 |
|-----------------------------------|--------|--------|---------------|----------|----------|-----------|
|
| 24 |
| **Open weights, closed datasets** | | | | | | |
|
| 25 |
| Qwen2-1.5B | 1.5B | ? | ❌ | 42.1 | **56.4** | **32.4** |
|
|
|
|
| 13 |
|
| 14 |
## Evaluation
|
| 15 |
|
| 16 |
+
We evaluate DCLM-1B using the [llm-foundry](https://github.com/mosaicml/llm-foundry) eval suite, and compare to recently released small models on key benchmarks.
|
| 17 |
As described in the paper, Core accuracy is the average of centered accuracy on
|
| 18 |
22 tasks (including HellaSwag and ARC-E), Extended is centered accuracy averaged
|
| 19 |
over 53 tasks.
|
| 20 |
|
| 21 |
|
| 22 |
+
| Model | Params | Tokens | Open dataset? | Core | MMLU 5-shot | Extended |
|
| 23 |
|-----------------------------------|--------|--------|---------------|----------|----------|-----------|
|
| 24 |
| **Open weights, closed datasets** | | | | | | |
|
| 25 |
| Qwen2-1.5B | 1.5B | ? | ❌ | 42.1 | **56.4** | **32.4** |
|