I wanted to try Qwen 3.8 Flash Next, a 177B MoE model, on the same RTX 3060 used in Part 1. Its quantized weights occupy 76.3 GiB, and my machine has 12 GB of VRAM and 32 GB of system RAM.
The first launch froze my desktop. Getting it working required changing how the server loaded the weights.
I used the UD-IQ3_XXS quantization and GenerelSchwerz’s llama-moe-cache branch to test GPU expert caching with weights backed by NVMe storage.