Boris-1.3-75M
Appearance
Boris-1.3-75M
Overview
Developer
Family
Boris-1.3
Type
Base (continued pretraining)
Fine-tuned from
License
Apache 2.0
Status
Released
Architecture
Architecture
GPT-2 decoder (pre-LayerNorm, learned positions, tied embeddings), 9 heads, dmodel 576
Parameters
77.4M
Layers
12
Context length
1,024 tokens
Tokenizer
GPT-NeoX-20B BPE, 50,304 (padded)
Training
Tokens
3.95B (1.55B base + 2.4B continued)
Tokens / param
51 Chinchilla-optimal ≈ 20
Data
Base: FineWeb-Edu 60% / DCLM 40%. Continued: DCLM-Baseline 1.5B, then FineWeb-Edu 0.3B + 0.6B
Hardware
1 × RTX 3060 12GB
Training time
14 h 49 m base + about 22 h continued
Compute
≈ 1.8 × 1018 FLOPs (6ND estimate)
Precision
bf16 autocast, fp32 master weights
Benchmarks
Benchmark
Score
Boris-75M
HellaSwag
27.57
27.20
ARC-Easy
40.57
39.14
ARC-Challenge
23.46
23.04
PIQA
59.41
57.18
WinoGrande
51.54
49.72
LAMBADA
18.16
15.21
Mean 6 tasks
36.8
35.2
Zero-shot; acc_norm for HellaSwag, PIQA and ARC, acc for WinoGrande and LAMBADA · Source: model card
Links
Weights
Formats
safetensors (transformers GPT-2)
Website
- Released3 months, 27 days ago
Boris base models
← Preceded by
Boris-250MSucceeded by →
Boris-1.3-125MBoris-1.3-75M is Boris-75M with 2.4B tokens of continued pretraining (one DCLM pass, then two FineWeb-Edu passes), aimed at the gaps in Boris-75M's own benchmark results.
External links
Image by Jakub Steiner / Wikimedia Commons / Licensed under CC BY-SA 2.0
This page or section is a stub. You can help the wiki by accurately contributing to it.