The Architecture of Precision: Variations in model quantizations
Summary of how quantization bridges the gap between trillion-parameter models and the hardware they run on, and why ‘smaller’ is almost always ‘faster’.
Inference
The Need For Speed: KV Cache and memory optimization at Inference
An introduction to KV Caching and its role in optimizing Transformer inference.