How I use AI/LLMs

It's worth noting down — for my own historical record, if anything — what my approach is to LLM usage and AI assisted coding, and how this has evolved over the months and (hopefully) years.

whisper.cpp / audio summarization

I began my exploration sometime this past summer (2026) when I wanted to build a tool to summarize our weekly scrums meeting through audio recording/parsing.

With the help of Gemini, I cobbled together a working prototype with whisper.cpp and some basic model to summarize the text.

Sufficiently impressed, I wanted to see whether this could apply to my day-to-day coding (spoiler alert; yes it does.)

~9B models

My main limitation was my setup. It was a decent setup for local inference, but has some limitations that I addressed going forward.

My daily driver is a Framework 13 from October 2023. It runs an AMD Ryzen 7640u with an on-die Radeon 760M chip. It shipped with 16GB memory, which back then I thought would be plenty… after all, I was upgrading from a computer with 8GB!

The main limitation was the memory. At the time, capable models that were able to fit on 16GB of VRAM were quite limited, and my attempts at finding one led me to smaller Gemma and Qwen models. While limited (and slow, and mistake-prone), they performed decently with sufficient context to work decently as an auto-complete aide. In nearly all cases, the time it took for the model to complete its work was longer than it would have been for me to do it myself.

MoE models

Around the time of Qwen 3.6 and Gemma 4 releases, I discovered that these models — now receiving many favourable reviews — were unable to run on my system because the model could not fit in unified memory in its entirety.

With the itch sufficiently scratched, I doubled my RAM (with a accordingly punishing $500 price tag!) to 32GB and was able to run these MoE models (Qwen 3.6 35B-A3B and Gemma 4 26B-A4B) easily. At this point, I also started using a TUI/harness called Pi and my day-to-day coding accelerated. I'm a developer that thinks through my problems, and my code writing is not as fluid as I'd like, so having a harness blitz through code generation was very freeing to me.

I opted for MoE (Mixture-of-Experts) models because I was able to achieve a blazing 15 tokens per second on my hardware, which finally reached the point where token generation matched my reading speed.

Gemma 4 ended up not working out because it used up too much memory and my system would OOM. Remember, this is a unified memory system meaning around 7-8GB was used by the OS itself. Moving everything non-LLM related to swap meant moderate slowdown across-the-board for anything else. Qwen 3.6 ended up serving much better as a coding agent with minimal memory usage for its K/V cache.

Dense models

Around the time of the release of the Qwen 3.8 line of models, I purchased an ASUS AMD AI PRO R9700. I connected this to my laptop using the AOOSTAR AG02 eGPU dock. My research led me to believe that this would be an acceptable tradeoff as most of the bottleneck is not due to the USB connection, but rather the memory bandwidth. So long as the model stayed in VRAM, any penalties would be minimized. This assumption turned out to be true.

I ended up making this purchase because the Qwen team did not release a small MoE in the 35B range. The comparative dense model would have a rough token generation speed of 4 tokens per second (woo!), with an MTP head bringing this up to 7. Workable but painful.

Swift Qwen

Qwen 3.8 reasons a lot. So much so that it became wholly unworkable to use 3.8 in real-time as a coding assistant. Luckily, the team at UkisAI released Swift Qwen, and later, Swift Qwen 1.5. A fine-tune of Qwen 3.8 that dramatically shortens its reasoning blocks by penalizing targeted terms that trigger thought-loops.

They explain it better than I:

We made Swift efficient by figuring out which tokens were linked to pathological overthinking and penalizing them without "attacking" the reasoning length directly then regained the accuracy with RL and OPD, leading to "compressed" token usage while maintaining accuracy.

With Swift Qwen 1.5 and the R9700, I am seeing prefill speeds of ~600 t/s, and a token generation speed of ~50 tokens per second.

← Back to all posts