Boris-75M
Appearance
Boris-75M
Overview
Developer
OpenCerebral (as New Millennium Artificial Intelligence)
Family
Boris-1
Type
Base (pretrained)
License
Apache 2.0
Status
Released; superseded by Boris-1.3-75M
Architecture
Architecture
GPT-2 decoder (pre-LayerNorm, learned positions, tied embeddings), 9 heads, dmodel 576
Parameters
77.4M
Layers
12
Context length
1,024 tokens
Tokenizer
GPT-NeoX-20B BPE, 50,304 (padded)
Training
Tokens
1.55B
Tokens / param
20 Chinchilla-optimal ≈ 20
Data
FineWeb-Edu 60% / DCLM-Baseline 40%
Hardware
1 × RTX 3060 12GB
Training time
14 h 49 m
Compute
≈ 7.2 × 1017 FLOPs (6ND estimate)
Precision
bf16 autocast, fp32 master weights
Benchmarks
HellaSwag
27.20
ARC-Easy
39.14
ARC-Challenge
23.04
PIQA
57.18
WinoGrande
49.72
LAMBADA
15.21
Mean 6 tasks
35.2
Zero-shot; acc_norm for HellaSwag, PIQA and ARC, acc for WinoGrande and LAMBADA · Source: Boris-1.3-75M model card
Links
Weights
Formats
safetensors (transformers GPT-2)
Website
- Released4 months, 5 days ago
Boris base models
← Preceded by
Boris-125MSucceeded by →
Boris-250MBoris-75M is a 77.4M-parameter base language model from OpenCerebral (then New Millennium Artificial Intelligence), trained from scratch on a single RTX 3060 for a Chinchilla-optimal 1.55B tokens.
External links
Image by Jakub Steiner / Wikimedia Commons / Licensed under CC BY-SA 2.0
This page or section is a stub. You can help the wiki by accurately contributing to it.