Running Qwen 3.6 35B NVFP4 Locally with FreeToken: Fitting a Coding MoE into 12GB VRAM [Part 1]

 Posted on September 11, 2026  | 

This is Part 1 of a three-part series on running, benchmarking, and scaling local coding agents. In Part 2, we benchmark this setup across 640 public trials and 280 private trials on a production Go monorepo, including Claude Code, Codex, Antigravity, local Qwen, and two hosted models. In Part 3, we build and test the specialized llama-moe-cache fork to run 177B Qwen3.8-Flash-Next on desktop hardware.

I wanted a capable local reasoning model for coding agents on my workstation, but my GPU is a standard desktop NVIDIA GeForce RTX 3060 with 12GB VRAM. The weights would not all fit in VRAM, so I needed a setup that could offload them.

After testing several options, I got Qwen 3.6 35B A3B NVFP4 running reliably with a full 64k token context window using FreeToken (ft), and hooked it up to both Crush and the Pi Coding Agent (pi). I ran into Hugging Face download pattern bugs, CUDA JIT compiler lookup failures, an aggressive 8k context ceiling that killed agent loops, and activation OOM spikes during long prefills.

These are the settings and local patches from that setup. Package versions can change; the commands below have not been rerun for this editorial revision.

The Hardware and Model

My workstation specs for this run:

  • GPU: NVIDIA GeForce RTX 3060 (12GB GDDR6 VRAM)

  • System RAM: 32GB DDR4

  • OS: Arch Linux (Kernel 6.13, CUDA 12.8 in /opt/cuda)

  • Storage: Fast NVMe SSD (/dev/nvme0n1)

A quick note on disk storage: when I first downloaded the weights, the Hugging Face cache lived on a mechanical SATA HDD. Cold starts took several minutes just reading the 21.8 GB safetensors files into memory. Moving the Hugging Face cache directory to a native NVMe SSD (HF_HOME=~/.cache/huggingface) reduced weight loading time to 8 seconds, with the entire server ready in ~55 seconds. For this setup, moving the weights to NVMe made startup much more practical.

Two earlier attempts failed on this machine:

  1. Qwen3.8-27B-FP8: This is a dense model, not an MoE. Even in FP8, holding 27B weights requires around 27GB of VRAM just to load. That attempt did not fit this setup.

  2. Qwen3.8-Flash-Next: An MoE model, but its expert offload architecture requires roughly 47.7 GiB of pinned host RAM. On a 32GB machine, host memory ran out instantly.

The model I got working was nvidia/Qwen3.6-35B-A3B-NVFP4. It is a 35B Mixture-of-Experts model, but only around 3B parameters are active per token (A3B). The weights are quantized to NVIDIA FP4 (NVFP4), totaling 21.8 GB on disk. FreeToken caches the base weights and the hot working set of experts in VRAM, streaming inactive experts from host RAM as needed.

Step 1: Setting Up the Python Environment

FreeToken works best in an isolated environment. I created a dedicated virtual environment with Python 3.13:

# Using virtualfish or standard venv
python -m venv ~/.virtualenvs/freetoken
source ~/.virtualenvs/freetoken/bin/activate

# Install PyTorch with CUDA 12.8 support and FreeToken
pip install --upgrade pip
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
pip install freetoken

Step 2: Fixing Startup and Memory Errors

The initial download and kernel compilation failed for different reasons.

1. Hugging Face Download Pattern Missing Config Files

When downloading nvidia/Qwen3.6-35B-A3B-NVFP4, FreeToken’s internal download helper used a restrictive pattern that grabbed .safetensors files but missed .json files. Without config.json and tokenizer metadata, the server crashed on startup.

To fix it, we patched ~/.virtualenvs/freetoken/lib/python3.13/site-packages/freetoken/utils/hf.py so snapshot_download includes *.json:

# In freetoken/utils/hf.py around line 211
return snapshot_download(
    model_path,
    allow_patterns=["*.safetensors", "*.json"],
    tqdm_class=DisabledTqdm,
)

2. Triton and NVCC Path Resolution

During model initialization, FreeToken compiles custom Triton kernels for the NVFP4 expert layers. On Arch Linux, the CUDA toolkit lives in /opt/cuda/bin, which was not in the system’s default $PATH. FreeToken failed with missing nvcc errors.

We resolved this with two targeted fixes:

First, create an nvcc wrapper in your local user path:

mkdir -p ~/.local/bin
cat << 'WRAPPER' > ~/.local/bin/nvcc
#!/bin/sh
exec /opt/cuda/bin/nvcc "$@"
WRAPPER
chmod +x ~/.local/bin/nvcc

Second, ensure the virtualenv’s ft launcher automatically injects CUDA_HOME and /opt/cuda/bin:

# Edit ~/.virtualenvs/freetoken/bin/ft right before main import
import os, sys
os.environ.setdefault("CUDA_HOME", "/opt/cuda")
if "/opt/cuda/bin" not in os.environ.get("PATH", ""):
    os.environ["PATH"] = f"/opt/cuda/bin:{os.environ.get('PATH', '')}"

3. PyTorch Memory Allocator Settings

To prevent VRAM memory fragmentation warnings and allocation failures, set the allocator configuration in your shell or profile:

export PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True"

4. The 8k Token Wall (Context Ceiling in Coding Agents)

This was the biggest functional issue. By default, FreeToken calculates how many KV cache pages to allocate based on remaining VRAM after loading weights. To keep a large MoE expert cache in GPU memory, it defaulted to only 8,204 tokens.

When running a real coding task in Crush or Pi, system instructions, tools, and repo context quickly reach 10,000 to 15,000 tokens. The agent immediately failed with:

Agent processing failed: failed to start agent processing stream: stream error:
prompt is too long: 13283 tokens > 8204 maximum (prompt + generation);
shorten the prompt or increase the KV cache budget.

The fix is to explicitly reserve the KV cache budget with --kv-reserve-tokens 65536.

5. Prefill Memory Spikes and Activation OOM

When you allocate 64k tokens of KV cache on a 12GB card, VRAM is tight. If an agent sends a 15k token prompt and the server tries to process the entire prefill in one forward pass, activation memory spikes and triggers CUDA Out-of-Memory.

We solved this with two parameters:

  • --memory-ratio 0.82: Allocates 82% of VRAM to static weights and caches, leaving ~1.7 GB of headroom for dynamic activations.

  • --max-prefill-length 4096: Chunks long prompt prefills into 4k token slices, capping activation memory spikes.

6. The 8k Output Token Cap

By default, FreeToken caps maximum generation output at 8,192 tokens. For simple chat completions that is plenty, but agent loops solving multi-file refactors or emitting detailed reasoning chains can easily exhaust an 8k output window in a single turn. When that happens, the model stops mid-sentence, leaving the agent with an incomplete patch or an empty response.

Adding --max-output-tokens 16384 doubles the headroom to 16k tokens.

The client examples below still request 8,192 output tokens. They document the baseline setup; raising the server limit alone does not make a client request 16k. Part 2 compares the baseline with low-thinking, 16k client configurations.

Step 3: Launching the FreeToken Server

This is the launch command I used:

ft serve \
  --model nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --moe-backend auto \
  --kv-reserve-tokens 65536 \
  --memory-ratio 0.82 \
  --max-prefill-length 4096 \
  --max-output-tokens 16384 \
  --port 1420

To avoid port collisions or accidentally starting duplicate instances, I wrapped this in a startup script (~/.local/bin/start-ft):

#!/usr/bin/env bash
set -euo pipefail

if ss -tulpn 2>/dev/null | grep -q ":1420\b"; then
  echo "FreeToken server is already running on port 1420."
  exit 0
fi

export HF_HOME="${HF_HOME:-$HOME/.cache/huggingface}"
export PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True"

exec ~/.virtualenvs/freetoken/bin/ft serve \
  --model nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --moe-backend auto \
  --kv-reserve-tokens 65536 \
  --memory-ratio 0.82 \
  --max-prefill-length 4096 \
  --max-output-tokens 16384 \
  --port 1420

On the very first launch, FreeToken builds the NVFP4 Triton expert banks in serial. This step takes around 5 to 7 minutes on an RTX 3060:

[core|rank=0] INFO expert banks: slow path (serial build)
Loading Qwen3.5 NVFP4 experts: 100%|██████████| 3/3 [06:48<00:00, 136.25s/it]
[core|rank=0] INFO NVFP4 expert backend: triton
[core|rank=0] INFO --moe-cache-auto resolved moe_cache_size=2283 num_pages=32781
[core|rank=0] INFO Allocating 65536 tokens for KV cache
[core|rank=0] INFO Free memory after initialization: 1.72 GiB
[core|rank=0] INFO Application startup complete. Uvicorn running on http://127.0.0.1:1420

Once running, verify the endpoint with a quick curl test:

curl -s http://127.0.0.1:1420/v1/models | jq .

Step 4: Configuring Crush CLI

Crush is an agentic coding CLI. To configure it for our FreeToken server, update ~/.config/crush/crush.json:

{
  "$schema": "https://charm.land/crush.json",
  "models": {
    "default": {
      "provider": "freetoken-local",
      "model": "qwen3.6-35b"
    }
  },
  "providers": {
    "freetoken-local": {
      "name": "FreeToken Local",
      "base_url": "http://127.0.0.1:1420/v1",
      "type": "openai-compat",
      "api_key": "local-development-bypass",
      "models": [
        {
          "id": "qwen3.6-35b",
          "name": "nvidia/Qwen3.6-35B-A3B-NVFP4",
          "context_window": 65536,
          "default_max_tokens": 8192,
          "can_reason": true,
          "supports_attachments": false
        }
      ]
    }
  }
}

This is the example prompt used to try the endpoint from Crush:

crush run "Write a bash script to check current memory usage"

Step 5: Configuring Pi Coding Agent

The Pi Coding Agent (pi) is a terminal coding agent that supports custom providers and models via ~/.pi/agent/models.json.

Here is the configuration to register FreeToken:

{
  "providers": {
    "freetoken": {
      "baseUrl": "http://127.0.0.1:1420/v1",
      "api": "openai-completions",
      "apiKey": "local-development-bypass",
      "compat": {
        "supportsDeveloperRole": false,
        "supportsReasoningEffort": false,
        "maxTokensField": "max_tokens"
      },
      "models": [
        {
          "id": "qwen3.6-35b",
          "name": "Qwen 3.6 35B A3B NVFP4",
          "reasoning": true,
          "input": ["text"],
          "contextWindow": 65536,
          "maxTokens": 8192
        }
      ]
    }
  }
}

A few important details for Pi:

  • apiKey: Pi requires an API key value to show the model as authorized in /model and --list-models, even though our local server does not enforce auth. A dummy string satisfies the check.

  • compat.supportsDeveloperRole: false: Ensures Pi sends instructions under the standard system role rather than the OpenAI developer role.

  • compat.supportsReasoningEffort: false: Prevents sending unsupported reasoning effort flags that FreeToken ignores or rejects.

Set it as the default startup model in ~/.pi/agent/settings.json:

{
  "theme": "dark",
  "defaultProvider": "freetoken",
  "defaultModel": "qwen3.6-35b"
}

Verify that Pi detects the model:

pi --list-models

You should see:

provider   model        context  max-out  thinking  images
freetoken  qwen3.6-35b  65.5K    8.2K     yes       no

Test it non-interactively:

pi -p "Write a python one-liner to print current date"

Pi completed the test through FreeToken. The benchmark in Part 2 covers longer tasks and the failures that appeared there.

Oh My Pi in the Benchmark

We also wired the same local FreeToken endpoint into Oh My Pi (omp). It used the OpenAI-compatible endpoint at http://127.0.0.1:1420/v1; tool and reasoning compatibility still depend on the client. In our benchmarks, we tested omp with low reasoning effort and the 16k output token cap.

Realistic Expectations and Caveats

Before relying on this as your daily driver, keep a few trade-offs in mind:

  • Startup time: The initial Triton compilation took around 6 minutes. The NVMe run above loaded weights in 8 seconds and had the server ready in about 55 seconds; weight loading is only part of startup.

  • Generation speed: Because inactive experts are paged from system RAM, generation speeds sit around 10 to 18 tokens per second on an RTX 3060. Treat that as a measurement from this setup, not a throughput guarantee for every prompt.

  • Keep context windows aligned: Make sure --kv-reserve-tokens on the server, context_window in Crush, and contextWindow in Pi all match (65536). If an agent assumes a 128k window while the server is capped at 64k, the server will reject long conversation branches.

Wrapping Up

I got nvidia/Qwen3.6-35B-A3B-NVFP4 running with a 64k context window on the 3060. The KV reservation and prefill limit were the settings that made the coding-agent prompts fit.

Crush, Pi, and Oh My Pi used this local model endpoint in the benchmark. Check each agent’s enabled tools separately if the whole workflow needs to stay offline.

Getting the server running left the question I wanted to test: how well would it edit code?

In Part 2: Benchmarking Terminal Coding Agents: 640 Public Trials and 280 Private Trials, we run an empirical benchmark across eight public tasks and sixteen agent configurations (640 trials), followed by 280 private trials inside a 15-package production Go monorepo to see where a 35B local model holds up, where it falls short, and what happens when terminal agents try to escape their workspaces.

Part 3 covers the llama-moe-cache experiment with the larger 177B model.

Categories: AI local-dev Linux 
Tags: llm qwen freetoken moe nvfp4 rtx-3060 local-ai crush pi-coding-agent arch-linux cuda