Running 177B MoE Models on a 12 GB GPU with llama-moe-cache and NVMe Streaming [Part 3]

I wanted to try Qwen 3.8 Flash Next, a 177B MoE model, on the same RTX 3060 used in Part 1. Its quantized weights occupy 76.3 GiB, and my machine has 12 GB of VRAM and 32 GB of system RAM.

The first launch froze my desktop. Getting it working required changing how the server loaded the weights.

I used the UD-IQ3_XXS quantization and GenerelSchwerz’s llama-moe-cache branch to test GPU expert caching with weights backed by NVMe storage.

The final launch command below uses a 32k context window and 48 GPU cache slots. Earlier notes used 64k and 20 slots; the client examples here have been aligned with the final command.

This post covers the build, memory failure, and recorded generation tests. The setup commands have not been rerun for this editorial revision.

The Model: Qwen 3.8 Flash Next

Qwen 3.8 Flash Next is an unusual architecture:

• 177 billion total parameters: A 125B parameter Mixture-of-Experts backbone combined with a 51B parameter N-gram phrasebook layer.

• 14 billion active parameters: Each token only activates a fraction of the total experts.

• Quantization: We use the Unsloth Dynamic IQ3_XXS quantization (UD-IQ3_XXS), split across three GGUF shards totaling 76.3 GiB (82 GB).

Only a subset of the experts is selected for each token. That makes caching selected expert weights worth testing.

How the Expert Cache Works

In standard llama.cpp GPU offloading, you decide how many full layers to place in VRAM and leave the rest in system RAM. How many layers fit depends on the model and the memory reserved for context and computation.

GenerelSchwerz’s moe-cache fork adds an expert cache:

1. Shared layers in VRAM: The embedding table, attention layers, and expert routing networks stay permanently in GPU VRAM.

2. GPU MoE Expert Cache: The GPU reserves a pool (48 slots in the final command below) managed with an LRU (Least Recently Used) eviction policy. When the router picks an expert, the engine checks if that slab is already sitting in the GPU cache. If it is a hit, the engine can use the cached weights.

3. NVMe Demand Paging: Inactive experts and the massive 51B phrasebook live on disk. When a cache miss occurs, the weights stream directly from the NVMe SSD over PCIe Gen4 into memory.

This turns your NVMe drive into an active storage tier for the model. But getting it to run safely on a machine with 32 GB of system RAM required learning a hard lesson about Linux memory allocation.

The Critical Gotcha: --load-mode mmap vs --load-mode none

When I first launched the server, I copied a launch script that passed --load-mode none.

That was a mistake.

Within twenty seconds of launching, my desktop froze. The mouse stuttered, audio stopped, and the system became completely unresponsive. The disk activity LED stayed solid on.

Here is what happened:

• --load-mode none tells llama.cpp to bypass memory mapping and allocate anonymous memory (malloc) for the model weights.

• On our machine with 32 GB of RAM, llama-server attempted to malloc all 76.3 GiB of GGUF weights into anonymous memory.

• The Linux kernel exhausted all 31 GB of physical RAM in seconds and dumped 40+ GB of pages into swap. The system began thrashing violently.

The original notes included a /proc/<pid>/status excerpt with RssAnon greater than VmRSS. Those values cannot describe one consistent resident-memory snapshot, so I have removed that excerpt pending a check against the original log.

I changed the launch command to use --load-mode mmap, as shown in Step 3.

When you use mmap, llama.cpp does not allocate anonymous memory for the weights. Instead, it maps the GGUF file from the NVMe SSD into address space. The kernel reads pages on demand and keeps them in the Linux page cache (RssFile).

Checking /proc/<pid>/status with --load-mode mmap:

VmSize:  129989116 kB
VmHWM:    27325948 kB
VmRSS:    27031160 kB
RssAnon:    286312 kB   <-- Only 286 MB of anonymous RAM!
RssFile:  26575968 kB   <-- File-backed clean pages from NVMe

Linux can reclaim clean file-backed pages without writing them to swap. The reported file-backed pages still occupy RAM while resident. They should not be counted as memory the model never used.

Step 1: Building llama-moe-cache with CUDA sm_86

First, install the prerequisites on Arch Linux:

sudo pacman -S base-devel cmake git cuda

Clone the repository and switch to the moe-cache branch:

cd ~/projects
git clone https://github.com/GenerelSchwerz/llama.cpp.git llama-moe-cache
cd llama-moe-cache
git checkout moe-cache

Configure and build with native optimizations and CUDA architecture 86 (matching Ampere cards like the RTX 3060):

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=86 \
  -DGGML_NATIVE=ON \
  -DCMAKE_CUDA_COMPILER=/opt/cuda/bin/nvcc

cmake --build build --target llama-server llama-cli -j8

Verify that build/bin/llama-server and build/bin/llama-cli were created cleanly.

Step 2: Downloading the Weights to NVMe

The model weights must live on a fast NVMe SSD. Running this over a SATA SSD or a spinning hard drive will bottleneck PCIe transfers down to 500 MB/s or less, making generation crawl.

Create the model directory on your NVMe partition:

mkdir -p ~/projects/llama-moe-cache/models/Qwen3.8-Flash-Next-GGUF/UD-IQ3_XXS
cd ~/projects/llama-moe-cache/models/Qwen3.8-Flash-Next-GGUF/UD-IQ3_XXS

Download the three shards using huggingface-cli:

huggingface-cli download unsloth/Qwen3.8-Flash-Next-GGUF \
  --include "UD-IQ3_XXS/*" \
  --local-dir ~/projects/llama-moe-cache/models/Qwen3.8-Flash-Next-GGUF/UD-IQ3_XXS \
  --local-dir-use-symlinks False

The three shards take 76.3 GiB total:

• Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf (27.9 GB)

• Qwen3.8-Flash-Next-UD-IQ3_XXS-00002-of-00003.gguf (27.9 GB)

• Qwen3.8-Flash-Next-UD-IQ3_XXS-00003-of-00003.gguf (26.1 GB)

Step 3: Configuring the Server Script

To fit comfortably within 12 GB of VRAM while maximizing expert cache residency, I used these settings:

1. -c 32768: Allocates a 32K token context window. While 64K is supported, right-sizing to 32K saves ~1.2 GB of VRAM, which we can directly reallocate to GPU expert cache slots.

2. -ctk q8_0 -ctv q8_0 -kvo: Quantizes both key and value KV caches to 8-bit. Standard FP16 KV cache would consume excessive VRAM. Quantizing to Q8 cuts KV memory in half.

3. -b 4096 -ub 512: Caps micro-batch size to 512 tokens, reducing transient CUDA workspace overhead.

4. -t 8: Locks CPU computation to 8 threads. The AMD Ryzen 7 5800X has 8 physical cores. This sets the thread count; it does not pin threads to particular cores.

5. --moe-expert-cache-size 48: Keeps 48 expert slabs cached directly on the GPU in VRAM (up from the conservative 20-slot baseline).

6. --lazy-mode on: Prevents preloading the entire 51B phrasebook into memory on boot.

7. --load-mode mmap: Uses clean file-backed mmap demand-paging, preventing system RAM exhaustion.

Here is the complete startup script (~/projects/llama-moe-cache/start-llama-moe.sh):

#!/usr/bin/env bash
set -euo pipefail

MODEL_DIR="/home/ann/projects/llama-moe-cache/models/Qwen3.8-Flash-Next-GGUF/UD-IQ3_XXS"
MODEL_FILE="$MODEL_DIR/Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf"
BINARY="/home/ann/projects/llama-moe-cache/build/bin/llama-server"
PORT="${LLAMA_PORT:-8080}"

if ss -tulpn 2>/dev/null | grep -q ":${PORT}\b"; then
  echo "llama-server is already running on port ${PORT}."
  exit 0
fi

if [ ! -f "$MODEL_FILE" ]; then
  echo "Error: Model file not found: $MODEL_FILE"
  exit 1
fi

echo "Starting llama-server (moe-cache) on port ${PORT} with Qwen 3.8 Flash Next..."
echo "Configuration: 8 CPU threads (physical cores), 48-slot GPU MoE cache, 32K context."

exec "$BINARY" \
  --model "$MODEL_FILE" \
  --host 127.0.0.1 --port "$PORT" \
  -ngl all -fit off \
  -c 32768 -b 4096 -ub 512 -np 1 \
  -t 8 \
  -fa on -ctk q8_0 -ctv q8_0 -kvo \
  --load-mode mmap \
  --lazy-mode on \
  --moe-expert-cache-size 48 \
  --cache-ram 0 \
  --alias qwen38-flash,Qwen3.8-Flash-Next,qwen3.8-flash \
  --jinja

And the companion stop script (~/projects/llama-moe-cache/stop-llama-moe.sh):

#!/usr/bin/env bash
set -euo pipefail

PORT="${LLAMA_PORT:-8080}"

echo "Stopping llama-server on port ${PORT}..."
pkill -f "llama-server.*--port ${PORT}" || true
pkill -f "llama-server.*Qwen3.8-Flash-Next" || true
sleep 1

if ss -tulpn 2>/dev/null | grep -q ":${PORT}\b"; then
  fuser -k "${PORT}/tcp" 2>/dev/null || true
fi

echo "llama-server stopped. VRAM released."

Make both executable and symlink them into ~/.local/bin/:

chmod +x ~/projects/llama-moe-cache/start-llama-moe.sh
chmod +x ~/projects/llama-moe-cache/stop-llama-moe.sh
ln -sf ~/projects/llama-moe-cache/start-llama-moe.sh ~/.local/bin/start-llama-moe
ln -sf ~/projects/llama-moe-cache/stop-llama-moe.sh ~/.local/bin/stop-llama-moe

Step 4: Connecting to Terminal Agents (Pi, Crush, and Oh My Pi)

llama-server exposes a standard OpenAI-compatible API on port 8080.

Configuring Pi (~/.pi/agent/models.json)

Add the llamacpp provider to your Pi configuration:

{
  "providers": {
    "llamacpp": {
      "baseUrl": "http://127.0.0.1:8080/v1",
      "api": "openai-completions",
      "apiKey": "local-development-bypass",
      "compat": {
        "supportsDeveloperRole": false,
        "supportsReasoningEffort": false,
        "maxTokensField": "max_tokens"
      },
      "models": [
        {
          "id": "qwen38-flash",
          "name": "Qwen 3.8 Flash Next 177B MoE",
          "reasoning": true,
          "input": ["text"],
          "contextWindow": 32768,
          "maxTokens": 16384
        }
      ]
    }
  }
}

Check that Pi detects the model:

pi --list-models

Check that the listed context matches the server’s 32,768-token allocation. The earlier captured output showed 65.5K and belonged to the older configuration.

Configuring Oh My Pi (~/.omp/agent/models.yml)

Add the corresponding block in ~/.omp/agent/models.yml:

providers:
  llamacpp:
    baseUrl: http://127.0.0.1:8080/v1
    api: openai-completions
    apiKey: local-development-bypass
    compat:
      supportsDeveloperRole: false
      supportsReasoningEffort: false
      maxTokensField: max_tokens
    models:
      - id: qwen38-flash
        name: Qwen 3.8 Flash Next 177B MoE
        reasoning: true
        input:
          - text
        contextWindow: 32768
        maxTokens: 16384

Configuring Crush (~/.config/crush/crush.json)

Add the llamacpp-local provider to your Crush configuration:

{
  "options": {
    "request_timeout": 600
  },
  "models": {
    "default": {
      "provider": "llamacpp-local",
      "model": "qwen38-flash"
    },
    "large": {
      "provider": "llamacpp-local",
      "model": "qwen38-flash"
    },
    "small": {
      "provider": "freetoken-local",
      "model": "qwen3.6-35b",
      "reasoning_effort": "low"
    }
  },
  "providers": {
    "llamacpp-local": {
      "name": "llama.cpp Local",
      "base_url": "http://127.0.0.1:8080/v1",
      "type": "openai-compat",
      "api_key": "local-development-bypass",
      "models": [
        {
          "id": "qwen38-flash",
          "name": "Qwen 3.8 Flash Next 177B MoE",
          "context_window": 32768,
          "default_max_tokens": 16384,
          "can_reason": true,
          "supports_attachments": false,
          "cost_per_1m_in": 0,
          "cost_per_1m_out": 0,
          "cost_per_1m_in_cached": 0,
          "cost_per_1m_out_cached": 0
        }
      ]
    }
  }
}

Verify that Crush recognizes the local endpoint:

crush models

Notice the "options": {"request_timeout": 600} setting. By default, Crush enforces a 60-second deadline before the first token arrives (LLM stream received no data for 1m0s). Because cold prompt evaluation on a 177B model across 1,500+ agent tokens can take 70 to 120 seconds before the first token is emitted, setting request_timeout: 600 prevents premature client-side aborts.

The Real Numbers

Once loaded, we ran generation benchmarks against the local endpoint:

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen38-flash",
    "messages": [
      {"role": "user", "content": "Write a python one-liner to calculate the factorial of 10."}
    ],
    "temperature": 0.0,
    "max_tokens": 512
  }' | jq .

These are the measurements recorded in the original experiment, not a fresh run. The system-memory figures mix available memory with reclaimable file cache, so they should not be read as the model’s total RAM footprint:

The model cleanly outputs reasoning tokens inside reasoning_content before delivering the final answer in content, matching the native DeepSeek/Qwen thinking format.

We also verified non-interactive execution directly from Pi:

pi -p --provider llamacpp --model qwen38-flash "Print 'HELLO_FROM_QWEN_177B' and nothing else."

In the server logs, Pi’s full 2,603-token agent prompt (including tool schemas and system instructions) evaluated at 10.82 tokens per second (240 seconds total). When the response returned, the follow-up turn reused 99% of the prefix via LCP cache matching, evaluating the new 23 tokens in just 4.27 seconds (5.38 t/s) before printing:

HELLO_FROM_QWEN_177B

Switching Between the Models

The recorded 177B decode tests reached 10.8 to 12.4 tokens per second, but Pi’s initial 2,603-token prompt took 240 seconds to evaluate. That wait matters in an interactive coding session.

I would need to run the coding benchmark on this model before claiming it is better at planning or debugging. The smoke test above only confirms that Pi can request a response.

The existing server scripts let me switch endpoints:

# Start the 35B setup
start-ft

# Switch to the 177B setup
stop-ft
start-llama-moe

# Return to the 35B setup
stop-llama-moe
start-ft

This assumes the local stop-ft helper is already installed; Part 1 only shows the startup script. Restart time depends on whether the weights are still in the OS page cache.

Wrapping Up

The fork ran the model on this machine and completed a Pi smoke test. That establishes that the setup can generate responses through an agent; it does not tell me how well the 177B model will solve the coding tasks from Part 2.

The settings I would check first when reproducing this setup:

• Use NVMe storage. SATA drives will bottleneck the expert streaming.

• Always use --load-mode mmap to prevent anonymous RAM exhaustion.

• Quantize the KV cache to 8-bit (-ctk q8_0 -ctv q8_0) to leave room in VRAM for the 32k context window and the 48-slot expert cache.

• The recorded command uses eight CPU threads. CPU affinity needs separate configuration.

This concludes our three-part local coding agent series:

• Part 1: Running Qwen 3.6 MoE Locally on an RTX 3060 with FreeToken, Crush, and Pi

• Part 2: Benchmarking Terminal Coding Agents: 640 Public Trials and 280 Private Trials

• Part 3: Running 177B MoE Models on a 12 GB GPU with llama-moe-cache and NVMe Streaming