report/report.md

nanochat training report

Generated: 2025-10-13 18:24:03

Environment

Git Information

• Branch: master
• Commit: 626bd3e (clean)
• Message: Add image of the WebUI to readme

Hardware

• Platform: Linux
• CPUs: 48 cores (96 logical)
• Memory: 1121.8 GB
• GPUs: 8x NVIDIA A100-SXM4-80GB
• GPU Memory: 634.0 GB total
• CUDA Version: 12.8
• Hourly Rate: $14.32/hour

Software

• Python: 3.10.12
• PyTorch: 2.8.0+cu128

Bloat

• Characters: 330,622
• Lines: 8,077
• Files: 42
• Tokens (approx): 82,655
• Dependencies (uv.lock lines): 2,004

Run started: 2025-10-13 18:24:07

---

Tokenizer training

timestamp: 2025-10-13 18:25:52

• max_chars: 2,000,000,000
• doc_cap: 10,000
• vocab_size: 65,536
• train_time: 89.5722
• num_special_tokens: 9
• token_bytes_min: 1
• token_bytes_max: 32
• token_bytes_mean: 6.9151
• token_bytes_std: 2.8736

Tokenizer evaluation

timestamp: 2025-10-13 18:26:00

Comparison with GPT-2

Text Type	Bytes	GPT-2 Tokens	GPT-2 Ratio	Ours Tokens	Ours Ratio	Relative Diff %
news	1819	404	4.50	375	4.85	+7.2%
korean	893	745	1.20	721	1.24	+3.2%
code	1259	576	2.19	493	2.55	+14.4%
math	1834	936	1.96	966	1.90	-3.2%
science	1112	260	4.28	225	4.94	+13.5%
fwe-train	4208518	900364	4.67	856901	4.91	+4.8%
fwe-val	4908443	1059062	4.63	1010356	4.86	+4.6%

Comparison with GPT-4

Text Type	Bytes	GPT-4 Tokens	GPT-4 Ratio	Ours Tokens	Ours Ratio	Relative Diff %
news	1819	387	4.70	375	4.85	+3.1%
korean	893	364	2.45	721	1.24	-98.1%
code	1259	309	4.07	493	2.55	-59.5%
math	1834	832	2.20	966	1.90	-16.1%
science	1112	249	4.47	225	4.94	+9.6%
fwe-train	4208518	874799	4.81	856901	4.91	+2.0%
fwe-val	4908443	1029691	4.77	1010356	4.86	+1.9%

Base model training

timestamp: 2025-10-14 01:32:23

• run: d0
• depth: 20
• max_seq_len: 2048
• num_iterations: -1
• target_flops: -1.0000
• target_param_data_ratio: 20
• device_batch_size: 32
• total_batch_size: 524,288
• embedding_lr: 0.2000
• unembedding_lr: 0.0040
• weight_decay: 0.0000
• matrix_lr: 0.0200
• grad_clip: 1.0000
• eval_every: 250
• eval_tokens: 10,485,760
• core_metric_every: 2000
• core_metric_max_per_task: 500
• sample_every: 2000
• model_tag:
• Number of parameters: 560,988,160
• Number of FLOPs per token: 3.491758e+09
• Calculated number of iterations: 21,400
• Number of training tokens: 11,219,763,200
• Tokens : Params ratio: 20.0000
• DDP world size: 8
• warmup_ratio: 0.0000
• warmdown_ratio: 0.2000
• final_lr_frac: 0.0000
• Minimum validation bpb: 0.8149
• Final validation bpb: 0.8149
• CORE metric estimate: 0.2059
• MFU %: 21.08%
• Total training flops: 3.917670e+19
• Total training time: 393.81m
• Peak memory usage: 75374.27MiB

Base model loss

timestamp: 2025-10-14 01:34:11

• train bpb: 0.8178
• val bpb: 0.8150
• sample 0: <|bos|>The capital of France is Paris. It is the largest city in France and the second largest in Europe.
• sample 1: <|bos|>The chemical symbol of gold is Au. It is a soft, malleable, ductile, and malleable metal. It
• sample 2: <|bos|>If yesterday was Friday, then tomorrow will be Saturday. If tomorrow is Sunday, then tomorrow will be Monday. If tomorrow is
• sample 3: <|bos|>The opposite of hot is cold. The opposite of cold is hot. The opposite of hot is cold.
• sample 4: <|bos|>The planets of the solar system are: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune,
• sample 5: <|bos|>My favorite color is red. I love the color red. I love the color red. I love
• sample 6: <|bos|>If 5x + 3 = 13, then x is 5 times 3. If 5x + 3 =

Base model evaluation

timestamp: 2025-10-14 01:41:37

• Model: base_model (step 21400)
• CORE metric: 0.1976
• hellaswag_zeroshot: 0.2598
• jeopardy: 0.0874
• bigbench_qa_wikidata: 0.5113
• arc_easy: 0.5354
• arc_challenge: 0.1183
• copa: 0.2800
• commonsense_qa: 0.0796
• piqa: 0.3798
• openbook_qa: 0.1627
• lambada_openai: 0.3839
• hellaswag: 0.2595
• winograd: 0.2821
• winogrande: 0.0513
• bigbench_dyck_languages: 0.1430
• agi_eval_lsat_ar: 0.1304
• bigbench_cs_algorithms: 0.3727
• bigbench_operators: 0.1762
• bigbench_repeat_copy_logic: 0.0312
• squad: 0.2389
• coqa: 0.2088
• boolq: -0.5218
• bigbench_language_identification: 0.1757

Midtraining

timestamp: 2025-10-14 02:01:41

• run: d0
• dtype: bfloat16
• max_seq_len: 2048
• device_batch_size: 32
• unembedding_lr: 0.0040
• embedding_lr: 0.2000
• matrix_lr: 0.0200
• init_lr_frac: 1.0000
• weight_decay: 0.0000
• final_lr_frac: 0.0000
• eval_every: 150
• eval_tokens: 10,485,760
• total_batch_size: 524,288
• Number of iterations: 765
• DDP world size: 8
• Minimum validation bpb: 0.4176

Chat evaluation mid

timestamp: 2025-10-14 02:16:06

• source: mid
• task_name: None
• dtype: bfloat16
• temperature: 0.0000
• max_new_tokens: 512
• num_samples: 1
• top_k: 50
• batch_size: 8
• model_tag: None
• step: None
• max_problems: None
• ARC-Easy: 0.3758
• ARC-Challenge: 0.2884
• MMLU: 0.3088
• GSM8K: 0.0303
• HumanEval: 0.0671
• ChatCORE metric: 0.0790

Chat SFT

timestamp: 2025-10-14 02:27:42

• run: d0
• source: mid
• dtype: bfloat16
• device_batch_size: 4
• num_epochs: 1
• max_iterations: -1
• target_examples_per_step: 32
• unembedding_lr: 0.0040
• embedding_lr: 0.2000
• matrix_lr: 0.0200
• weight_decay: 0.0000
• init_lr_frac: 0.0200
• eval_every: 100
• eval_steps: 100
• eval_metrics_every: 200
• Training rows: 20,843
• Number of iterations: 651
• Training loss: 1.2206
• Validation loss: 1.0725

Chat evaluation sft

timestamp: 2025-10-14 02:39:37

• source: sft
• task_name: None
• dtype: bfloat16
• temperature: 0.0000
• max_new_tokens: 512
• num_samples: 1
• top_k: 50
• batch_size: 8
• model_tag: None
• step: None
• max_problems: None
• ARC-Easy: 0.3952
• ARC-Challenge: 0.2961
• MMLU: 0.3138
• GSM8K: 0.0402
• HumanEval: 0.0549
• ChatCORE metric: 0.0870

Summary

• Characters: 330,622
• Lines: 8,077
• Files: 42
• Tokens (approx): 82,655
• Dependencies (uv.lock lines): 2,004

Metric	BASE	MID	SFT	RL
CORE	0.1976	-	-	-
ARC-Challenge	-	0.2884	0.2961	-
ARC-Easy	-	0.3758	0.3952	-
GSM8K	-	0.0303	0.0402	-
HumanEval	-	0.0671	0.0549	-
MMLU	-	0.3088	0.3138	-
ChatCORE	-	0.0790	0.0870	-

Total wall clock time: 8h15m