Boris-250M
Appearance
Boris-250M
Overview
Developer
OpenCerebral (as New Millennium Artificial Intelligence)
Family
Boris-1
Type
Base (pretrained)
License
Apache 2.0
Status
Released
Architecture
Architecture
GPT-2 decoder (pre-LayerNorm, learned positions, tied embeddings), 18 heads, dmodel 1,152
Parameters
250.4M
Layers
12
Context length
1,024 tokens
Tokenizer
GPT-NeoX-20B BPE, 50,304 (padded)
Training
Tokens
5.01B
Tokens / param
20 Chinchilla-optimal ≈ 20
Data
FineWeb-Edu 60% / DCLM-Baseline 40%
Hardware
1 × RTX 3060 12GB
Training time
105 h 56 m (about 4.4 days)
Compute
≈ 7.5 × 1018 FLOPs (6ND estimate)
Precision
bf16 autocast, fp32 master weights
Benchmarks
HellaSwag
33.7
ARC-Easy
47.7
ARC-Challenge
26.1
PIQA
64.3
WinoGrande
50.0
LAMBADA
31.0
Mean 6 tasks
42.1
Zero-shot; acc_norm for HellaSwag, PIQA and ARC, acc for WinoGrande and LAMBADA. Published to one decimal, in the card's chart · Source: model card
Links
Weights
Formats
safetensors (transformers GPT-2)
Website
- Released3 months, 30 days ago
Boris base models
← Preceded by
Boris-75MSucceeded by →
Boris-1.3-75MBoris-250M is a 250.4M-parameter base language model from OpenCerebral (then New Millennium Artificial Intelligence), the largest of the Boris-1 models, trained from scratch on a single RTX 3060 for a Chinchilla-optimal 5.01B tokens.
External links
Image by Jakub Steiner / Wikimedia Commons / Licensed under CC BY-SA 2.0
This page or section is a stub. You can help the wiki by accurately contributing to it.