Parameter Golf Research:
AI-Assisted ML Competition

What Is Parameter Golf?

Parameter Golf is an OpenAI research competition where the objective is to train a language model that achieves the lowest possible bits-per-byte (BPB) score on a held-out evaluation set — subject to strict constraints:

• Training time: 10 minutes on 8×H100 SXM GPUs
• Evaluation time: 10 minutes on 8×H100 SXM GPUs
• Total file size: ≤16 MB (code + compressed weights combined)
• Statistical validity: 3-seed average required for p<0.01 significance

BPB (bits per byte) measures how well a model predicts text. Lower is better. The SOTA baseline at the start of this campaign was 1.0810 BPB.

Results Summary

The strongest single-seed result shown here is 1.21925538 BPB from PR #1974, and the latest submission is 1.15865 BPB as a 3-seed mean in PR #1983. Neither PR has been reviewed, merged, or officially scored by OpenAI.

Competition Submissions

Two pull requests were opened against the openai/parameter-golf repository under the track_10min_16mb track.

Submission Snapshots

PR #1974 — Details Verified In The Submission Body

Architecture summary from PR #1974

11-layer transformer, 512d hidden, 8 attention heads (4 KV heads)
SP8192 vocabulary (SentencePiece BPE, 8192 tokens)
Depth recurrence: layers 3-5, NUM_LOOPS=2
Parallel residuals: layers 7+
TTT: 1 epoch, LR=0.005, momentum=0.9, chunk=32k tokens
SDCLIP: 20 steps
GPTQ int6 quantization + Brotli compression
Result: val_bpb 1.21925538, best seed 42

PR #1983 — Currently Verified From The Public PR

Infrastructure

Required NCCL Configuration (8×H100)

export NCCL_SOCKET_IFNAME=eth0
export NCCL_IB_DISABLE=1
export NCCL_P2P_DISABLE=1
export NCCL_CUMEM_ENABLE=0
export NCCL_NVLS_ENABLE=0

These settings are required to prevent communication deadlocks on multi-GPU distributed training. Omitting any one of them can cause silent hangs.

RunPod Operational Lessons

The 8×H100 pod is where your final submission run happens — it is not the place to be figuring out credentials, tooling, or file management. Every minute the pod is alive costs money. These are the hard lessons from running this campaign.

Pre-Pod Checklist

# Verify SSH key is loaded and reachable
ssh -i ~/.ssh/id_ed25519_runpod -o BatchMode=yes root@<pod-ip> -p <port> echo "auth ok"

# Verify GitHub SSH auth
ssh -T git@github.com

# Verify HuggingFace token (if pulling datasets)
huggingface-cli whoami

# Verify dataset is present and shards are healthy
find ~/data -name "*.bin" | wc -l
find ~/data -name "*.bin" -size -100M -exec echo "SUSPECT: {}" \;

Post-Run Save Sequence

# After every run — before touching the pod stop button
cp /tmp/train_seed42.log ~/saved_logs/
cp /tmp/train_seed1337.log ~/saved_logs/
cp /tmp/train_seed314.log ~/saved_logs/

# Push logs and submission artifacts to GitHub immediately
cd ~/parameter-golf-fork
git add records/ submission.json
git commit -m "save: final run logs seed 42/1337/314"
git push origin your-submission-branch

# Only terminate the pod AFTER the push confirms success

Campaign Timeline

Ablation Study Results

Top 5 Configurations by BPB

Key Findings

Seed Variance

Verified Submission Metrics

All metrics above are taken directly from the submission body of PR #1974 and PR #1983.

What Didn't Work

Documenting failures is as important as documenting wins.

EMA Blend at Eval

Idea: At evaluation, replace pure EMA weights with a blended checkpoint. Theory: EMA over-smooths, base under-smooths, blend is optimal.

Result: ~−0.02 BPB regression after quantization. The EMA blend interacts badly with GPTQ. Pre-quant gain disappears and reverses post-quant.

Auto-Calibrated Training Schedule

Useful for reducing cross-machine variance (~0.008 BPB σ reduction) but irrelevant for single-machine final submission. Engineering cost not justified.

Broad Cross-Machine Sweeps

Cross-machine variance (σ ~0.027) overwhelmed any signal from narrow hyperparameter differences (σ ~0.003–0.010). Local node sweeps were more reliable than cross-machine averaging.

High MATRIX_LR (≥0.042)

Consistently regressed vs lower settings on the local node. The optimal band was confirmed at 0.010–0.015. The high-LR assumption was a false inheritance from early ARM node results where batch-scaling shifted the effective LR.

Lessons Learned

1. Validate post-quantization, not pre. Every experiment that showed gains pre-quantization had to be re-validated after GPTQ. Several techniques performed well before compression and worse after. The final eval metric is post-quant BPB.
2. 3-seed averaging is non-negotiable. The σ ~0.018 seed variance means a single-seed result can mislead by up to ±0.036 BPB at 2σ. Single-seed directional signal is fine; single-seed submission decisions are not.
3. Know your noise floor before designing sweeps. Cross-machine σ ~0.027. Seed σ ~0.018. Effect sizes below 2σ are undetectable. This eliminated most speculative ideas early.
4. Local validation is cheaper than you think. A 3,000-step local node run (~12 min) provides directional signal reliable enough to make hyperparameter decisions before committing to H100.
5. Hardware diversity reduces bottlenecks. Running ARM node sweeps in parallel with local node validations meant the sweep pipeline was never bottlenecked on a single machine.
6. The byte budget interacts with everything. At 16 MB total (code + compressed weights), a technique that adds 0.5 MB might be worth it at 0.010 BPB gain but not at 0.002. Always check the byte budget.
7. One shot means prioritize reliability over peak. With only one H100 run for submission, use the configuration most likely to land well across 3 seeds — not the highest-ceiling configuration.

Reproducibility

Shard Verification

# Verify shard sizes — corrupt shards silently degrade BPB
find ~/data -name "*.bin" -size -100M -exec echo "SUSPECT: {}" \;
# Any shard under ~100MB should be investigated

Training Command (Generic)

python3 -u train_gpt.py \
  --MATRIX_LR=0.010 \
  --SCALAR_LR=0.030 \
  --ITERATIONS=20000 \
  --WARMDOWN_ITERS=3000 \
  --MAX_WALLCLOCK_SECONDS=600 \
  --TTT_EPOCHS=1 \
  --TTT_LR=0.005 \
  --TTT_CHUNK_SIZE=32768 \
  --SDCLIP_STEPS=20 \
  --seed=42

Final Submission Command (8×H100)

export NCCL_SOCKET_IFNAME=eth0
export NCCL_IB_DISABLE=1
export NCCL_P2P_DISABLE=1
export NCCL_CUMEM_ENABLE=0
export NCCL_NVLS_ENABLE=0

torchrun --nproc_per_node=8 train_gpt.py \
  --MATRIX_LR=0.010 \
  --SCALAR_LR=0.030 \
  --TTT_EPOCHS=1 \
  --TTT_LR=0.005 \
  --SDCLIP_STEPS=20 \
  --seed=42

What's Next

The next step is straightforward: iterate on the strongest documented levers from the PR #1974 and PR #1983 branches.

High Confidence

• SDCLIP on all submission seeds — the PR #1974 body documents a +0.0081 BPB improvement from SDCLIP on seed 42 (1.2273→1.2192). Low-risk, well-validated.
• Best-of-K seed selection — running K=5 seeds and submitting the best individual seed gains approximately 0.013 BPB in expectation.

Speculative but Promising

• Quant-aware EMA (N3) — track rounded weights in EMA during final 10% of training. Directly targets the pre→post quant gap.
• Post-quant temperature calibration (N9) — scan temperature τ ∈ {0.97..1.03} on val after quantization, bake argmin into submission. 0.002–0.005 BPB expected.
• Model soup (N2) — average final weights of 3 seeds before submission. Reduces seed variance σ 0.018→0.010 at K=3.