Choosing a GPU for Running LLMs at Home

If you take one thing from this guide: buy the most VRAM you can afford.

Why VRAM dominates

Model size, context length, and batch size all consume memory. Run out, and performance collapses as data spills to system RAM. A card with more memory but slightly slower cores will beat a faster card that cannot fit your model.

• Entry: enough VRAM for quantized mid-size models
• Sweet spot: a card that fits popular open models comfortably
• Enthusiast: multi-card or high-memory workstation GPUs

Quantization stretches your VRAM, but it is not a substitute for having enough in the first place.

Beyond the card

Do not forget power supply headroom and case airflow. A starved or thermally throttled GPU quietly loses you performance.