PetitGPT: Training and Evaluating a 124.6M Language Model on One GPU

I trained and released PetitGPT, a 124.6M-parameter language model built from scratch on a single RTX 4090. The project now includes a custom tokenizer, approximately 13 billion pretraining token positions, instruction tuning, published benchmark results, and downloadable weights.

Building the full pipeline gave me a way to study how architecture, data, optimization, and evaluation interact under a small compute budget. The released model reaches 57.74% accuracy on ARC-Easy and 28.16% on ARC-Challenge. Its pretraining validation loss reaches 2.4702. The experiments also show how improvements in reference prediction or a narrow instruction task can coexist with weaker performance elsewhere.

This article describes the research-v1 release and the selected alpha075 checkpoint. The technical report contains the full measurements and protocols; the Hugging Face repository contains the model and native inference bundle.

The released model

Parameters

124,635,456 unique parameters

Decoder

30 layers · width 576

Attention

9 query heads · 3 key/value heads

Feed-forward network

SwiGLU · width 1,536

Context and tokenizer

2,048 tokens · 32k byte-level BPE

Pretraining

Approximately 13B token positions

Hardware

One NVIDIA RTX 4090

Release

alpha075 · native PyTorch CUDA inference

A deeper, narrower decoder

The released architecture uses 30 Transformer blocks with a hidden width of 576. Each block combines RMSNorm, grouped-query attention, rotary position embeddings, and a SwiGLU feed-forward network. There are nine query heads and three key/value heads, each with dimension 64. Input embeddings and the output head share one weight matrix.

This distributes the parameter budget across more layers while keeping individual layers compact. Grouped-query attention shares keys and values across query heads, and tied embeddings avoid a second 32,000 × 576 vocabulary matrix. The exact parameter count is 124,635,456, with the shared matrix counted once.

The tokenizer is a custom 32,000-token byte-level BPE model with seven control tokens. It applies no text normalizer. Chat boundaries are inserted by token ID, so a literal special-token spelling inside user content is treated as ordinary text. These choices keep source text and conversation structure distinct throughout training and inference. Architecture and tokenizer details are in the model specification.

What improved during pretraining

Pretraining ran for 49,590 optimizer updates at a 2,048-token sequence length. Micro-batches of eight sequences with 16 accumulation steps gave an effective batch of 128 sequences. A Muon/AdamW optimizer combination and one warmup–stable–decay learning-rate schedule spanned both data stages.

Stage A selected about 10B serialized tokens from FineWeb-Edu, DCLM-Edu, FineWiki English, and Python-Edu. Stage B selected about 3B more, adding structured tutorials, PES2O, and StackExchange while retaining the earlier source families. The optimizer actually processed 12,999,720,960 input positions; selected, packed, and processed token counts are reconciled separately in the report.

Reference validation loss fell from 2.7486 at the end of Stage A to 2.4702 at the end of Stage B, with final perplexity 11.83. All seven reference source families improved. The largest absolute drops were in structured tutorials (0.6189), StackExchange (0.4710), and PES2O (0.3282), the three families added in Stage B.

This is a measured improvement over the course of training. Data mixture, additional token exposure, and the learning-rate schedule changed together, so the experiment does not isolate their individual contributions. The useful result is that the same reference validation tracked progress across both stages, including the new source families. See the pretraining results for the per-source measurements and logged timing windows.

Instruction tuning and the selected blend

I used two post-training stages. P2 fine-tuned the base on 12,000 concise instruction conversations for 750 updates, supervising every assistant turn and its ending token. P3 adapted it to copying, field lookup, set membership, and JSON tasks, with replay examples from the earlier instruction data.

P3 increased passes on the 512-item procedural development battery from 4 under P2 to 484 at step 320. At the same time, loss on the original P2 validation conversations rose from 1.322110 to 1.400923. The model became better at the procedural tasks while assigning less probability to the earlier reference answers.

I then evaluated fixed weight blends between P2 and P3 step 320. The released alpha075 uses 75% of the displacement toward P3. It reached 487/512 procedural passes and a validation loss of 1.365311, recovering some of P2's reference likelihood. The 0.50 blend preserved more reference likelihood but reached only 445/512 passes, below the study's 90% development target.

Alpha075 was a useful working reference for this trade-off. The development battery contains correlated, repeatedly inspected items, and another strict generation diagnostic fell from 20/100 for P3 step 320 to 16/100 for the blend. I therefore keep these scores separate from general assistant quality and from the later public benchmark campaign. The post-training analysis records both the gains and the losses.

Results under a shared evaluation protocol

I evaluated the released weights alongside SmolLM-135M-Instruct and SmolLM2-135M-Instruct. All numbers below were measured in this project on pinned revisions, using the same task rows and numerical protocol for each model.

PetitGPT leads both comparators on the two ARC tasks and trails them on PIQA and HellaSwag. Scoring uses raw completion prompts, each model's own tokenizer, FP32 computation, and no chat template or answer generation. The models differ in training data, compute, and architecture; no significance test or contamination audit was performed. These results describe performance under the recorded protocol.

I also ran IFEval, a generation-based instruction-following evaluation with 541 prompts and 834 instructions. Prompt-level strict accuracy was 17.19% for PetitGPT, 10.35% for SmolLM, and 21.63% for SmolLM2. Each model used its native chat formatting, greedy generation, and a 1,280-new-token cap, with programmatic scoring and no reviewing language model. The separate IFEval results show substantial instruction-following limitations for all three models.

What generation still gets wrong

Multiple-choice accuracy measures how a model ranks candidate answers. Free-form generation also requires producing the right facts, preserving source content, following constraints, and completing an implementation. I kept those questions separate in a versioned review of 189 prompts per model, with actual outputs retained for inspection.

The Python results make the distinction concrete: PetitGPT produced the requested function interface on 42 of 46 prompts, while none passed the whole-answer review. In one example,count_matches_regex returned the matches rather than their count. In another, average_or_default had the expected signature but never used its default argument. The published cases also include successful summarization and dialogue responses.

These are small, reused diagnostics with model-assisted judgments and explicitly recorded owner clarifications. The 0/46 figure is a whole-answer review outcome, not training accuracy or a uniform unit-test pass rate. The success and failure cases make the scoring boundaries visible instead of leaving the reader with only an aggregate number.

Later adaptation experiments brought some local gains alongside losses in other capabilities. DPO, response distillation, and LoRA did not produce a replacement that met the recorded retention criteria. The released weights remain the P2/P3 blend. Those experiments are documented in the research results.

Making the experiment inspectable

The release includes the weights, tokenizer, configuration, native PyTorch inference code, source snapshots, and a public training workflow. It also includes machine-readable result tables, the plotting data behind both figures, evaluation protocols, and selected generated answers. Readers can inspect the measurements and the training path behind the selected model.

Export validation checked all 213 named state entries and 60 rotary buffers. Eight source/export fixture pairs matched within their respective numerical profiles, including generated tokens and stopping behavior. This establishes parity in the measured environment. The released CLI requires CUDA and supports native BF16 and FP32 profiles; a Transformers loading path is not implemented.

The public new-run workflow accepts separately supplied data and creates new runs. It checks tokenizer identity, packed shard inventories, assistant supervision counts, and deterministic update plans. Original data and private evaluation populations are not included, so following the recipe with new inputs does not reproduce the published scores automatically.

• Download the model and tokenizer
• Run native inference
• Train and evaluate a new model
• Read the technical report and evidence tables

What I would investigate next

The project has progressed from a training stack to a released model with measured results. I can now trace improvements through pretraining, compare post-training checkpoints on common diagnostics, and show what the model actually generates. That provides a much firmer starting point for the next experiment.

My next priorities would be fresh evaluation data fixed before candidate selection, repeated runs for the most promising comparisons, and controlled instruction-data scaling. The central question remains how to improve useful generation while preserving capabilities already learned.

Source: PetitGPT research-v1 technical report and its accompanying tables, figures, and examples. This article summarizes the released results; the report preserves the full measurement scope.