Boris-1.7-D60M-n30M
Appearance
Boris-1.7-D60M-n30M
Overview
Developer
Family
Boris-1.7
Type
Base (pretrained, experimental)
License
Apache 2.0
Status
Released
Architecture
Architecture
Decoder-only transformer: 12 heads / 4 KV heads (GQA), SwiGLU 1920, RoPE, RMSNorm, dmodel 768, 30M n-gram table after block 2
Parameters
116.1M (60.0M dense + 25.2M embedding + 30.0M n-gram table)
Layers
10
Context length
1,024 tokens
Tokenizer
Own byte-level BPE, 32,768
Training
Tokens
2.5B
Tokens / param
21.5 Chinchilla-optimal ≈ 20
Data
Ultra-FineWeb bulk; Cosmopedia v2 + FineWeb-Edu anneal over the final 15%
Hardware
1 × RTX 3060 12GB
Training time
22 h 37 m (10,172 steps)
Compute
≈ 1.3 × 1018 FLOPs (6ND estimate), n-gram table excluded
Precision
bf16, torch.compile
Benchmarks
HellaSwag
31.22
ARC-Easy
45.88
ARC-Challenge
26.02
PIQA
62.57
WinoGrande
51.07
LAMBADA
26.66
ArithMark-3
36.1
BananaMind Base Bench 1.1
1022 Elo
Mean 6 tasks
40.6
Zero-shot; acc_norm for HellaSwag, PIQA and ARC, acc for WinoGrande and LAMBADA. ArithMark-3 is left out of the mean · Source: model card
Links
Formats
safetensors (custom loader, not transformers)
Website
- Released2 months, 22 days ago
Boris base models
← Preceded by
Boris-1.3-125MSucceeded by →
Boris-2-Preview-1Boris-1.7-D60M-n30M is an experimental base model: a 60M-parameter dense transformer with a 30M-parameter n-gram lookup table spliced into the residual stream, testing whether 30M of cheap lookup buys what 30M of dense weight would at a fixed 2.5B-token budget.
External links
Image by Jakub Steiner / Wikimedia Commons / Licensed under CC BY-SA 2.0
This page or section is a stub. You can help the wiki by accurately contributing to it.