🎯 GPT-Neo Enhanced SPG: 450x Compression with FULL Non-Negotiables Compliance

GPT-Neo Capabilities:

  • Max Sequence Length: 2048 tokens (full 2048 context)
  • Optimal Datasets: wikitext, openwebtext, pile, c4

Available Models:

  • GPT-Neo 125M: 12 layers, suitable for quick testing
  • GPT-Neo 1.3B: 24 layers, balanced size/performance
  • GPT-Neo 2.7B: 32 layers, largest open GPT-Neo model

STRICT COMPLIANCE MODE:

  • ✅ NO hardcoding - All from config
  • ✅ NO estimations - Measured only
  • ✅ NO fallbacks - Fail fast
  • ✅ NO fake results - Reproducible
  • ✅ Clean code - Full validation
  • ✅ Hardware validation - GPU memory checked
GPT-Neo Model Variant
Compression Methods
128 2048
5 50
1 5
Dataset
0.85 0.99
0.5 5
5 50
5 50
10 200
100 500
100 500
Head Retention
Magnitude Threshold

GPT-Neo Specific Settings:
1 48
0 8

405x+ Compression Settings (adjusted for GPT-Neo):
0.0001 0.001
0.0001 0.001
1 64
0 1
0.0001 0.01
1 16
1 10
50 600
100 1000

Compression Trade-off Analysis:
3 8

GPT-Neo 450x Compression Results

🔬 GPT-Neo Architecture Details

Model Specifications:

  • GPT-Neo 125M: 12 layers, 768 hidden dim, 12 heads
  • GPT-Neo 1.3B: 24 layers, 2048 hidden dim, 16 heads
  • GPT-Neo 2.7B: 32 layers, 2560 hidden dim, 20 heads
  • Maximum Context: 2048 tokens (full 2048)

Memory Requirements:

  • 125M: Minimum 1GB VRAM
  • 1.3B: Minimum 6GB VRAM
  • 2.7B: Minimum 12GB VRAM (16GB+ recommended)

Optimal Datasets for GPT-Neo:

  • WikiText: Clean Wikipedia articles
  • OpenWebText: High-quality web text (GPT-2 training data recreation)
  • The Pile: 800GB diverse text corpus
  • C4: Colossal Clean Crawled Corpus

Compression Adjustments for GPT-Neo:

  • Adjusted stage compression ratios for architecture
  • Optimized recent window for layer count
  • Reserved FP16 heads tuned per model size
  • Memory cleanup for 2.7B model
  • Full 2048 token context support

📦 Proving Protocol Features

Attestable Proof Bundle (.zip) contains:

  • Full environment and configuration
  • Per-sample raw measurements
  • Layer-level compression fingerprints
  • Exact package versions for reproducibility

Verification:

  • Recomputes summary from raw records
  • Validates compression ratio achievement
  • Checks numerical tolerances
  • Hard-fails in CI if verification fails

This ensures research-grade reproducibility on GPT-Neo models with full 2048 token context.