OSCAR: 2-Bit KV Cache Quantization for LLMs

What's New in OSCAR

OSCAR (Offline Spectral Covariance-Aware Rotation) represents a fundamental shift in how KV cache quantization works for long-context language models. Rather than applying generic data-oblivious transforms, this system learns attention-specific rotation patterns offline to preserve the most critical information.

• Attention-Aware Rotation: Derives separate rotations for keys and values from covariance structures estimated during offline analysis, capturing which dimensions matter most for attention computation
• 2-Bit Quantization: Compresses KV cache to 2.28 bits per element, achieving extreme compression while maintaining accuracy within 1-4 points of full precision baselines
• 8× Memory Reduction: Cuts KV cache memory footprint to one-eighth of original size, enabling larger batch sizes and longer context windows on existing hardware
• 3× Decode Speedup: Accelerates token generation by up to 3× at 100K context length, directly improving latency for real-time applications
• Open-Source Release: Fully available for community use, integration, and further research without proprietary restrictions
• Minimal Accuracy Loss: Achieves only 3.78-point gap on Qwen3-4B-Thinking and 1.42-point gap on Qwen3-8B compared to BF16 baseline

Technical Specifications

OSCAR implements sophisticated quantization through attention-aware spectral analysis and offline rotation computation. The system operates at the infrastructure level, optimizing how transformer models store and retrieve cached key-value pairs during inference.

• Quantization Precision: INT2 format at 2.28 bits per KV element, compared to 32-bit or 16-bit floating-point baselines
• Rotation Computation: Offline spectral covariance analysis generates separate rotation matrices for keys and values, applied during inference without runtime overhead
• Context Window Support: Tested and optimized for 100K token context lengths, supporting modern long-context model requirements
• Model Compatibility: Validated on Qwen3-4B-Thinking-2507 and Qwen3-8B, with architecture applicable to other transformer-based LLMs
• Memory Bandwidth: Reduces KV cache memory bandwidth requirements proportionally to compression ratio, enabling faster memory access patterns

Official Benefits

• 8× KV Memory Reduction: Compresses key-value cache to one-eighth original size, freeing GPU memory for larger models or batch sizes
• Up to 3× Decode Speedup: Accelerates token generation latency by 3× at 100K context, directly improving end-user experience for real-time applications
• Minimal Accuracy Degradation: Only 1.42-3.78 point accuracy gap versus full precision, maintaining model quality while achieving extreme compression
• Offline Computation: Rotation matrices computed once offline, eliminating runtime quantization overhead and enabling seamless integration
• Cost Reduction: Enables long-context serving on smaller GPUs or fewer instances, directly reducing infrastructure costs for production deployments

Real-World Translation

What Each Feature Actually Means:

• Attention-Aware Rotation: Instead of blindly compressing all dimensions equally, OSCAR learns which parts of the KV cache matter most for attention calculations. In practice, this means a chatbot handling 100K-token conversations can compress memory without losing the ability to reference important context from earlier in the conversation.
• 2-Bit Quantization: Your model's cached data gets squeezed to 2 bits per value instead of 16 or 32 bits. For a production system running 1000 concurrent requests with 100K context each, this transforms an impossible memory requirement into something that fits on standard GPUs.
• 8× Memory Reduction: A GPU that previously could handle 2 concurrent long-context requests now handles 16. This directly translates to lower infrastructure costs and better resource utilization in production environments.
• 3× Decode Speedup: Users waiting for responses see answers arrive 3× faster. For customer-facing AI applications, this improvement in latency directly impacts user satisfaction and reduces perceived lag.
• Offline Rotation Computation: The system learns optimal compression patterns once during setup, then applies them instantly during inference. This means zero additional latency overhead compared to uncompressed serving.

Getting Started

How to Access

• GitHub Repository: Visit Together AI's GitHub to clone the OSCAR repository and access the complete source code, documentation, and implementation examples
• Installation: Install via pip or clone the repository directly, with dependencies listed in requirements.txt for Python environments
• Model Integration: Download pre-quantized model weights or quantize existing models using OSCAR's provided scripts and configuration files
• Documentation: Review the official documentation for API reference, integration guides, and performance tuning parameters specific to your hardware

Quick Start Guide

For Beginners:

1. Clone the OSCAR repository and install dependencies using pip install -r requirements.txt
2. Download a compatible model (Qwen3-4B or Qwen3-8B) and place it in the models directory
3. Run the quantization script with default parameters: python quantize.py --model qwen3-4b --output ./quantized_models
4. Test inference with the provided benchmark script to verify speedup and memory reduction on your hardware

For Power Users:

1. Analyze your specific model's attention patterns using the offline covariance analysis tool to generate custom rotation matrices
2. Configure quantization parameters in the YAML config file, adjusting bit precision, rotation computation settings, and hardware-specific optimizations
3. Integrate OSCAR into your inference serving framework (vLLM, TensorRT-LLM, or custom CUDA kernels) using the provided integration examples
4. Benchmark against your baseline using the performance profiling tools, measuring memory usage, latency, and accuracy degradation on your specific workloads
5. Deploy to production with monitoring hooks to track quantization effectiveness and adjust parameters based on real-world performance data

Pro Tips

• Batch Size Optimization: With 8× memory savings, increase batch size proportionally to maximize GPU utilization and throughput on your hardware
• Context Window Tuning: Test OSCAR on your actual context lengths (not just 100K) since compression effectiveness may vary based on attention patterns in your specific use cases
• Accuracy Validation: Run your model on representative samples from your actual data distribution before production deployment, as accuracy gaps may differ from published benchmarks
• Hardware Profiling: Profile memory bandwidth and compute utilization on your specific GPU to identify bottlenecks and confirm the 3× speedup applies to your infrastructure