Yang Qi

PetitGPT

A 124.6M Language Model Trained from Scratch

• Technical write-up→
• Repository (opens in a new tab)
• Model weights (opens in a new tab)

A released language model pretrained on approximately 13B token positions using one RTX 4090, with a custom tokenizer, instruction tuning, reproducible evaluation, and native PyTorch inference.

• Built a 30-layer, 124.6M-parameter decoder with grouped-query attention, RMSNorm, SwiGLU, and tied embeddings; pretraining reached reference validation loss 2.4702.
• Reached 57.74% ARC-Easy and 28.16% ARC-Challenge accuracy, ahead of two evaluated SmolLM 135M instruct baselines under the same protocol; results were lower on PIQA and HellaSwag.
• Published the weights, tokenizer, training recipes, evaluation protocols, loss curves, and success/failure cases; studied adaptation and capability retention through SFT, DPO, response distillation, and LoRA.

Model size

124.6Mreleased parameters

Pretraining

13Btoken positions

Training hardware

1×RTX 4090

Efficient Tensor Approximation

Alternating CNN

A neural architecture for compressing high-dimensional CSI tensor data through efficient approximation of low-rank tensor structure.

• Developed a neural architecture for high-dimensional CSI tensor data compression.
• Designed the approach to efficiently approximate low-rank tensor structure.
• Achieved normalized reconstruction error below 0.01 and reduced training time by approximately 25% relative to a 3D-CNN baseline.

Normalized reconstruction error

< 0.01achieved

Training time

≈25% lowerrelative to 3D-CNN baseline

Reference

3D-CNNbaseline

Correlated Spiked Tensor Models

High-dimensional recovery

• arXiv paper (opens in a new tab)

Theory and algorithms for recovering multiple correlated spikes in high-dimensional tensor models.

• Studied detection phase transitions and local-optimization accuracy using random matrix theory and high-dimensional optimization.
• Derived statistical guarantees for correlated latent components.
• Derived an asymptotically unbiased estimator of signal strength.