Boris-125M
Appearance
Boris-125M
Overview
Developer
OpenCerebral (as New Millennium Artificial Intelligence)
Family
Boris-1
Type
Base (pretrained)
License
Apache 2.0
Status
Released; superseded by Boris-1.3-125M
Architecture
Architecture
GPT-2 decoder (pre-LayerNorm, learned positions, tied embeddings), 12 heads, dmodel 768
Parameters
124.5M
Layers
12
Context length
1,024 tokens
Tokenizer
GPT-NeoX-20B BPE, 50,304 (padded)
Training
Tokens
2.5B
Tokens / param
20.1 Chinchilla-optimal ≈ 20
Data
FineWeb-Edu 60% / DCLM-Baseline 40%
Hardware
1 × RTX 3060 12GB
Training time
33 h 39 m
Compute
≈ 1.9 × 1018 FLOPs (6ND estimate)
Precision
bf16 autocast, fp32 master weights
Benchmarks
HellaSwag
29.33
ARC-Easy
41.75
ARC-Challenge
23.89
PIQA
59.74
WinoGrande
49.72
LAMBADA
22.86
Mean 6 tasks
37.9
Zero-shot; acc_norm for HellaSwag, PIQA and ARC, acc for WinoGrande and LAMBADA · Source: Boris-1.3-125M model card
Links
Weights
Formats
safetensors (transformers GPT-2)
Website
- Released4 months, 7 days ago
Boris base models
Succeeded by →
Boris-75MBoris-125M is a 124.5M-parameter base language model, the first model released by OpenCerebral (then New Millennium Artificial Intelligence). It was trained from scratch on a single RTX 3060.
External links
Image by Jakub Steiner / Wikimedia Commons / Licensed under CC BY-SA 2.0
This page or section is a stub. You can help the wiki by accurately contributing to it.