Boris-2-Preview-2
Appearance
Boris-2-Preview-2
Overview
Developer
Family
Boris-2
Type
Base (pretrained, preview)
License
Apache 2.0
Status
Released; superseded by Boris-2
Architecture
Architecture
Decoder-only transformer: 8-head MHA, SwiGLU 1664, RoPE, Canon layers, 15M n-gram table
Parameters
143.7M
Layers
31 (dmodel 512)
Context length
2,048 tokens
Tokenizer
Own byte-level BPE, 32,768
Training
Tokens
102.5B (100B stable + 2.5B decay)
Tokens / param
713 Chinchilla-optimal ≈ 20
Data
Main: ClimbMix ~90%, UltraData-Math L3 10%, Orca-Math 0.15% (FineMath before 55B). Decay adds Cosmopedia tutorials, MetaMathQA, Calc-ape210k/mawps, DCLM
Hardware
1 × NVIDIA V100 32GB (~59,500 tok/s)
Compute
≈ 7.9 × 1019 FLOPs (6ND estimate), n-gram table excluded
Precision
fp16 mixed, fp32 residual stream
Benchmarks
Benchmark
Score
Preview-1
HellaSwag
46.60
45.23
ARC-Easy
59.13
58.38
ARC-Challenge
30.20
29.95
PIQA
70.84
69.21
ArithMark-3
42.20
37.60
Mean 4 tasks
51.7
50.7
Zero-shot, lm-evaluation-harness 0.4.12, acc_norm. ArithMark-3 is left out of the mean · Source: model card
Links
Formats
safetensors (transformers, trust_remote_code)
Website
- Released1 month, 1 day ago
Boris base models
← Preceded by
Boris-2-Preview-1Succeeded by →
Boris-2Boris-2-Preview-2 is the second preview of Boris-2: the main run's 100B-token checkpoint with a 2.5B-token learning-rate decay, the same decay length as Boris-2-Preview-1.
External links
Image by Jakub Steiner / Wikimedia Commons / Licensed under CC BY-SA 2.0
This page or section is a stub. You can help the wiki by accurately contributing to it.