Boris-1.3-125M
Appearance
Boris-1.3-125M
Overview
Developer
Family
Boris-1.3
Type
Base (continued pretraining)
Fine-tuned from
License
Apache 2.0
Status
Released
Architecture
Architecture
GPT-2 decoder (pre-LayerNorm, learned positions, tied embeddings), 12 heads, dmodel 768
Parameters
124.5M
Layers
12
Context length
1,024 tokens
Tokenizer
GPT-NeoX-20B BPE, 50,304 (padded)
Training
Tokens
5.16B (2.5B base + 2.66B continued)
Tokens / param
41.4 Chinchilla-optimal ≈ 20
Data
Base: FineWeb-Edu 60% / DCLM 40%. Continued: FineWeb-Edu (1.81B over 5 passes) and DCLM-Baseline (1.6B over 2 passes)
Hardware
1 × RTX 3060 12GB
Training time
33 h 39 m base + about 37 h continued (estimated)
Compute
≈ 3.9 × 1018 FLOPs (6ND estimate)
Precision
bf16 autocast, fp32 master weights
Benchmarks
Benchmark
Score
Boris-125M
HellaSwag
29.68
29.33
ARC-Easy
42.51
41.75
ARC-Challenge
24.23
23.89
PIQA
61.32
59.74
WinoGrande
52.72
49.72
LAMBADA
25.79
22.86
Mean 6 tasks
39.4
37.9
Zero-shot; acc_norm for HellaSwag, PIQA and ARC, acc for WinoGrande and LAMBADA · Source: model card
Links
Weights
Formats
safetensors (transformers GPT-2)
Website
- Released3 months, 22 days ago
Boris base models
← Preceded by
Boris-1.3-75MSucceeded by →
Boris-1.7-D60M-n30MBoris-1.3-125M is Boris-125M with about 2.66B tokens of continued pretraining in seven passes, leading with FineWeb-Edu before DCLM to avoid the ARC regression seen in Boris-1.3-75M.
External links
Image by Jakub Steiner / Wikimedia Commons / Licensed under CC BY-SA 2.0
This page or section is a stub. You can help the wiki by accurately contributing to it.