The goal
Globirie summarises everything it ingests, and none of it goes to a cloud API. Ingestion runs around 4,000 articles a day from 158 active feeds out of 412 registered, and the corpus is past 800,000 articles in a 28 GB Postgres database. The no-cloud rule is self-imposed and it is most of what makes the project interesting to run.
What fits on 28 GB
The GPU node has an RTX 4070 Ti SUPER with 16 GB and an RTX 4070 SUPER with 12 GB. Qwen3.6 35B A3B at Q4_K_XL is roughly 20 GB of weights, so it loads on neither card alone and has to straddle both, with whatever is left going to the KV cache. llama.cpp serves it with a 65k context across four slots.
One model does both jobs. It answers assistant questions over the corpus and calls tools, and it also runs the structured enrichment pass that gives every article a summary, key points, countries and an importance score.
The mixture of experts part is doing the heavy lifting
A3B means about 3 billion active parameters per token out of 35 billion resident. You pay for the large model in VRAM and get throughput closer to a small one, and that is the only reason a single 35B can serve interactive traffic and grind through bulk enrichment at the same time. The best single day was 125,395 articles enriched, against an ingestion rate of about 4,000 a day. A dense model of that size on these cards would not be worth running.
What the logs think is running
This one fooled me, so it gets its own section.
The extraction config names a model, and every enriched row stores that name
in an llm_model_used column. Neither means anything. llama.cpp serves
whatever GGUF it was started with and ignores the model field in the request
entirely: ask it for gpt-4 and it will answer from the Qwen weights it has
loaded, echoing the GGUF path back at you as the model name.
So that column records what the pipeline asked for, not what answered. Mine
went on naming a small 4B long after the box had been switched over to the
35B, and the only way to know what is actually loaded is to ask the server
itself at /props. If you are going to log a model name, log the one the
server reports back rather than the one you sent.
Articles are longer than you want them to be
Extraction chunks at 8,192 characters with 1,024 characters of overlap, and each chunk summary is capped at 1,024 tokens. The workload is lopsided: long text in, short structured JSON out. Prefill dominates, so batching and lean prompts are what help, while clever decoding tricks do very little. There is a Qwen-tuned prompt variant that replaced the generic baseline for exactly that reason, and a grammar retry for the times the model returns JSON that will not validate.
Thinking mode, and why it is off
The best failure I have had here. Qwen routes its reasoning into a separate
reasoning_content field. With thinking enabled, a long think can consume the
entire output cap and return a response containing no visible content at all.
It does not error and it does not time out. The call simply succeeds and comes
back empty, which I watched happen live on cluster chats in August.
So the assistant runs with thinking off, which also streams more crisply. The showcase path keeps it on, because there the calls are short enough that turning it off starves them of reasoning budget instead. So it is one model running two opposite settings, and response length is what decides which one applies.
One box, in the end
Extraction started out pointing at a pool of llama.cpp servers with per-endpoint slot counts, the idea being that load could move without a redeploy and the gaming machine could join when its GPU was idle. In practice the pool collapsed to a single box. One host is retired now and everything runs on the GPU node, reached over a WireGuard link.
The machinery was not wasted, because being able to move load by editing one variable is what made retiring that host painless. The honest description is still one machine doing all the work.
What I would change
The 2080 Ti in the other machine is fully occupied with embeddings and reranking, so it cannot help here. What I want is a single 24 GB card to replace the awkward split, since pushing a model across a PCIe link costs latency that one card would not. I have not bought one. Nothing on the market is at a price I am willing to pay, so the two-card arrangement stays until that changes.