Floating Point to Integer Quantization
Normally, weight values are stored as FP16 (2 bytes per parameter). A 7B parameter model requires 14GB of VRAM just to load. By quantizing parameters to 4-bit integers, we reduce VRAM to 3.5GB.
Normally, weight values are stored as FP16 (2 bytes per parameter). A 7B parameter model requires 14GB of VRAM just to load. By quantizing parameters to 4-bit integers, we reduce VRAM to 3.5GB.
What is GGUF format best suited for?
Tap card to flipCPU inference with partial GPU offloading, commonly used for running local LLMs on Macs/PCs.
Auto-saves immediately
Review core concepts before doing the quiz
What is GGUF format best suited for?
Tap card to flipCPU inference with partial GPU offloading, commonly used for running local LLMs on Macs/PCs.
Always online
Hi! I'm Spooky, your study buddy! Let's learn together.