This service handles retrieval embeddings and listwise reranking over HTTP, with OpenAI-compatible endpoints. Both models stay loaded in one process and share a GPU. High-priority search requests can preempt index-time embedding, so bulk work does not block live search.
Globirie queues newly enriched articles for embedding, and its assistant uses the stored vectors to find related coverage. The globe’s story clusters are built separately from article metadata and time proximity.
The interesting part: one GPU, two workloads
Requests are routed into lanes by intent. input_type: document goes to
the bulk lane at normal CUDA priority, queries and reranks ride the
interactive lane at high priority and preempt bulk work mid-flight. The
server owns batching entirely: it coalesces concurrent requests and groups
by length, which is why clients sending their own batch sizes only make
things slower.
Measured instead of vibes
I evaluate quality through the live HTTP endpoints with 1,271 BEIR queries from SciFact, NFCorpus and FiQA. The report uses paired query-level confidence intervals. Reranking lifts aggregate nDCG@10 by +0.0127, and the report says plainly where it doesn’t: significant gains on FiQA, significant losses on SciFact recall and F1. Embedding sustains ~105 documents/second while serving; a top-50 listwise rerank lands in ~450 ms p50 on FiQA.