Globirie watches hundreds of news feeds, pulls article text, summarises it with locally-hosted models, and groups related coverage into stories you can explore on a globe or ask an AI assistant about. It has gathered over 800,000 articles so far, a few thousand more every day.
What it does
Feeds are polled on adaptive schedules with conditional requests. New articles enter an extraction queue, get full-text extracted, then a structured LLM enrichment pass adds summaries, key points, countries, and importance scores. A deterministic clustering pass groups related coverage into stories and anchors them geographically for the globe. It matches article metadata: title tokens, keywords, entities, issue and conflict fields, geography, and publication times. Embeddings from the RAG Backend support the assistant’s separate related-article search.
Engineering decisions
Clustering is deterministic on purpose
Stories are grouped by a periodic batch pass over recent articles. Shared metadata limits the candidate pairs, then weighted token overlap and time proximity decide which articles join a story. Geography contributes to the score and the globe’s anchor without requiring every member to name the same country.
With the same article metadata, feed authority scores and settings, the pass is reproducible. When a cluster looks wrong I can inspect the signals that joined its members and run the same inputs again.
Identity needs separate handling because each rebuild can change a story’s membership. The materializer compares membership overlap with the previous snapshot and carries forward matching cluster ids, so new coverage can join a story while its globe links keep working. It also records splits, merges and retirements. Deterministic grouping makes the result inspectable; overlap matching preserves identity as the corpus changes.
Why the queue lives in Postgres
Work items live in the articles table with a lock id, an attempt counter, a next-attempt timestamp and a failure cap. Workers claim rows instead of consuming from RabbitMQ or Celery.
The reason is the hardware. These are home machines and they are not dependable the way a datacentre is. I tinker with something and take a box down, or the power dips, and whatever was in flight dies with it. A broker would have been one more service to keep alive on exactly the machines that keep falling over.
Claiming a row stamps a fresh lock id and bumps the attempt count, and completion is guarded by that same lock id, so a worker that outlived its lease cannot overwrite the result from whoever reclaimed the item. A claim older than two hours is treated as crashed and becomes reclaimable. That timeout has to comfortably exceed the longest legitimate chunked extraction, or healthy work gets stolen out from under itself. Anything that exhausts its three attempts is marked terminal rather than looping forever, and the failure is classified as an access gate, a bot block, empty content or something transient.
Delivery is at-least-once, not exactly-once. A crash between a handler’s side effect and its completion re-runs that handler later, so handlers have to be idempotent. That is the tradeoff for not running a broker.
The other thing a database queue buys is a backlog you can interrogate. I can ask SQL what is pending, what keeps failing and why, which is how I found that 1,692 articles were being rejected for falling between 123 and 299 characters against an old 300-character floor. Most of them were genuine short wire briefs, so the floor moved to 150.
Feeds settle on their own polling rate
Every feed carries its own interval, somewhere between one minute and thirty. A poll that returns new or changed content resets that feed to one minute. A 304, or a poll with nothing new in it, multiplies the interval by 1.5, and an error doubles it. Because the requests are conditional, checking a quiet feed costs almost nothing.
The effect is that the active feeds sort themselves out with no hand-tuned schedule. Wire services that publish constantly sit near the floor, and a blog that posts twice a month drifts out to the ceiling and stays there until it has something to say.