Fine-Tuning a Local LLM to Write in My Voice: QLoRA on an RTX 3060 with Unsloth

I use big LLMs to draft blog posts and LinkedIn posts. They help with research and structure, but I still spend time rewriting them to sound like me. Every draft comes out with the same polished, slightly corporate voice, and even after I edit it by hand, something of that voice stays in the text. There is actually research confirming this: a 2026 study on post-editing found that even when humans edit LLM drafts, the result stays measurably closer to LLM style than to the author’s own writing.

So I decided to test an idea: keep the big model for the heavy lifting (research, structure, first draft), and fine-tune a small local model to do one job only – rewrite the draft in my voice. This post is about what it took to get there on a single RTX 3060, and everything that broke along the way.

Why I Tried Fine-Tuning

Before spending a weekend on it, I checked whether fine-tuning on a few hundred examples has any chance. These papers gave me reasons to try it:

• LIMA showed that ~1,000 curated examples are enough to align a model, because small-data fine-tuning teaches style and format, and knowledge stays in the base model.

• Panza built personalized email assistants from under 100 emails. In their blind study, over two thirds of generated emails were rated as credibly human.

• TinyStyler beat GPT-4 on authorship style transfer with an 800M parameter model.

• A tone-of-voice study got a 1B model to match a target style with only 100 samples, beating system-prompting baselines.

Those studies use different tasks and evaluation methods, so they do not tell me how many examples my rewriter will need. First I needed to find enough of my own writing to test it.

Step 1: Finding My Own Writing

I found much less clean blog prose than I expected. My blog has posts going back to 2011, but everything from 2023 onward was written with AI assistance to some degree, so it can’t be ground truth for my voice. After stripping code blocks, the clean 2015-2018 posts came out to about 2,900 words of prose. That left me with very little blog material for this experiment.

So I went digging for places where I actually type:

• Gmail Takeout. An 11GB mbox parsed with Python’s built-in mailbox module. Keep only messages where I am the sender, strip quoted text and signatures, keep English messages with 60+ words of real prose. After filtering: ~15,000 words.

• Beeper. Beeper’s desktop app keeps all bridged chats in a local SQLite database (~/.config/BeeperTexts/index.db), with a convenient isSentByMe flag. My work Slack turned out to be the best source here. One trick that tripled the yield: merging message bursts – consecutive messages I sent in the same chat within 5 minutes – into a single sample, because I tend to type several short lines instead of one long one. ~15,500 words.

• Agent session logs. Claude Code and Codex keep full session transcripts as JSONL on disk. My own typed prompts in there are authentic first-person writing too. After filtering out pasted documents and code: ~11,000 words.

Corpus extraction and QLoRA training ran locally. Data preparation also included hosted claude -p calls to generate neutral drafts, as described below, so the full workflow was not local-only.

One warning if you try this with your own mailbox: automated garbage is everywhere. My first pass happily included Google Calendar invitation boilerplate as “my writing”. Filter aggressively.

Step 2: Creating Training Pairs

A rewriter model needs pairs: a bland draft as input, my text as output. I have the outputs. Nobody wrote boring versions of my emails to serve as inputs.

The trick is to generate the inputs backwards, and it has a name in the literature: inverse paraphrasing (STRAP), or “reverse instructions” in the Panza paper. For every piece of my real writing, a fixed model (always the same one, for consistency) rewrites it as a neutral, personality-free draft with the same facts. Then the pair goes into training reversed: neutral draft in, my original out.

A real example from my corpus. My 2015 blog post said:

Long story short: … I tested it personally and it worked like a charm.

The de-styled draft the model trains against:

To summarize the process: … This was tested directly and worked without issue.

I built roughly 150 pairs this way, with each neutral draft paired with my original wording.

The pairs are formatted as chat SFT – system prompt says “you are Ann’s voice rewriter”, user message carries the draft plus a register tag (blog / email / chat), assistant message is my original. Held-out split is by document, so no chunk of a test document ever leaks into training.

Step 3: Training on the 3060

The stack is Unsloth + TRL’s SFTTrainer, QLoRA at rank 16 on all attention and MLP projections. The base model question had a hardware answer:

• Gemma-3-12B was my first pick. In 4-bit it needs ~7GB of VRAM for weights alone, and with my desktop session already holding 2.5GB of the card’s 12GB, the loader refused – bitsandbytes will not train with layers offloaded to CPU. I did not get it to load alongside that desktop session.

• Qwen3-8B in 4-bit sits at ~5.5GB and trains comfortably next to a running desktop.

With ~150 pairs, 5 epochs and early stopping on held-out loss, a full training run takes 16-20 minutes on the 3060. That number changed how I worked: at 20 minutes per experiment you stop being precious about runs and start iterating like it’s a unit test suite.

This is the adapter configuration from the training script. It assumes the model is already loaded; it is not a standalone training program:

model = FastModel.get_peft_model(
    model,
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    use_gradient_checkpointing="unsloth",
)
# in SFTConfig: learning_rate=1e-4 (more on that below),
# per_device_eval_batch_size=1, prediction_loss_only=True

The last two settings mattered in this run: evaluation exhausted VRAM while converting full logits to fp32.

What Broke During Training

The draft generator inherited plan mode. I generated the de-styled drafts with headless claude -p calls. What I did not notice: the calls inherited the project’s plan mode, and 48 of 173 drafts began with meta-commentary like “This is a simple text rewriting task, not a coding task…”. The model trained on that junk, and run 1 would sometimes respond to a rewrite request with reasoning about the task instead of the rewrite. Lesson: read your generated training data. All of it, or at least a real random sample.

Run 2 copied the input. After cleaning the data I added a “keep the same length, do not add content” instruction and kept a conservative learning rate of 1e-5. The result faithfully returned the input, word for word. Copying was the cheapest way to satisfy the loss. Raising the LR to 1e-4 and softening the instruction fixed it.

The OOM killer, twice. The kernel killed training at the exact moment weight loading finished. There was plenty of free VRAM, so I stared at CUDA for a while before checking free -h: a browser was eating 10GB of host RAM and zram swap was already 60% full. If your training dies with no Python traceback right after the weights load, look at host memory first.

Boilerplate contamination. Calendar invites, HTML-heavy forwarded newsletters, and pasted AI output inside my own emails all made it into early corpus versions. Each one needed its own filter.

Results So Far

After four runs, email and chat examples looked more promising than blog prose. These were my own judgments, not a blind evaluation.

Email and chat improved in the examples I inspected. A de-styled draft comes in as “Hello, apologies for the delayed response – this week was quite busy… Before running it, execute npm install” and comes out as “Hello, sorry for the delay, this week was very busy… Before running it, run npm install to install the dependencies.” Lowercase tech terms, comma splices, no corporate connectors. Facts intact. It reads like me on a normal day.

Blog drafts mostly stayed unchanged. The model mostly returns blog drafts untouched. One likely reason is the training mix: only 20 of ~150 training pairs are blog register, because that’s all the clean blog prose I have. The fix in progress is hand-curating my AI-assisted 2023+ posts back into my real voice, to produce pairs of AI drafts and my own edits, closer to the rewriting task I want the model to perform.

Two caveats I want on record. First, sounding familiar to me does not establish that other readers would recognize my voice. Research on forensic stylometry asks a different question from the editing help I want here. Second, I have not yet run the honest benchmark against my existing setup (a detailed style guide plus few-shot examples in the big model’s prompt). Practitioner reports are split on whether fine-tuning beats strong prompting, so the deciding test is a blind comparison where I rank outputs without knowing which is which. If the fine-tune only ties, the writeup will say so.

Wrapping Up

The pipeline works end to end: corpus extraction from mail, chat and agent logs, pair synthesis by de-styling, QLoRA training in 20-minute iterations on a 12GB consumer card. Some email and chat outputs sound closer to me in my own review. The register I actually started this for – blog posts – is still waiting on better data. As usual, most of the real work turned out to be data work.

Next steps: finish curating the blog pairs, retrain, run the blind eval, and if the fine-tune earns its keep, wire it into my drafting workflow as a local llama-server behind the big model.

Training ran on Arch Linux with an RTX 3060 12GB and 32GB RAM. Model weights and checkpoints live on a separate drive because the home partition filled up mid-project.. but that’s another story.