Skip to content

Project

← Back

Opinion Search

RUNNING

A search engine over 19,187 EV discussions from Reddit, YouTube comments and EV publications. It uses BM25 and vector retrieval for search, then labels aspect sentiment with a trained RoBERTa cascade. Its RAG summaries cite the retrieved text.

  • Solr
  • Qdrant
  • RoBERTa
  • Qwen3 Embedding
  • Cross-encoder reranking
  • ABSA
  • Flask
  • React

19,187

documents

4.06M

words

0.8610

subjectivity macro F1 · RoBERTa ep4

~7.1×

speedup · 30-doc rule-gated sample

F1 is from the four-epoch RoBERTa benchmark under the fixed 70/15/15 protocol. Speedup compares an earlier rule-gated hybrid with routing every document to an LLM on a 30-document serial sample. The later 19,187-document labeling pass is a separate experiment.

Watch a real query travel the pipeline

Recorded runs from the system's own benchmark, with the documents and measurements exactly as it produced them.

pipeline replay RECORDED RUN
  1. spellcheck recovery
  2. BM25 · Solr
  3. dense retrieval · Qdrant
  4. cross-encoder rerank
  5. sentiment join · RoBERTa cascade

demo video coming soon

A search engine for what people actually think about electric vehicles, built over 19,187 cleaned documents (4.06M words) crawled from ~40 subreddits, YouTube comments and EV publications.

How retrieval works

Three tiers: BM25 over Solr with EV-domain synonyms and spellcheck recovery, dense retrieval over chunk-level Qwen3 embeddings in Qdrant, and a hybrid tier that fuses both candidate pools and reranks the top 16 with a cross-encoder.

The classification cascade

Sentiment isn’t computed at query time. An offline cascade (rules → SenticNet → fine-tuned RoBERTa → LLM escalation, confidence-gated at every hop) labels the whole corpus, then labels are joined onto results from the store. The final 19,187-document pass resolved every document: rules handled 848, RoBERTa handled 13,550, and the LLM handled 4,789, or 24.96% of the corpus.

Engineering decisions

Span extraction did not survive contact with forum text

The original plan was classic ABSA: locate the opinion target as a span in the text, then classify the sentiment attached to that span. I built it with PyABSA and it worked in the narrow sense that it ran and reproduced. The spans it returned were too noisy to put in front of anyone.

The problem is the writing. A product review has tidy targets sitting right there in the sentence. Somebody arguing about EVs on Reddit spreads the opinion across a whole paragraph, or buries it in an aside, or implies it without ever naming the thing, and often there is simply no contiguous span that is the target. Ask a model to point at one anyway and you get a confident pointer at nothing in particular.

So the annotation scheme moved to fit how people actually write. Documents get classified into aspect families (charging, infrastructure, range, performance, cost, maintenance_reliability) along with what they think about them, rather than having a location extracted. That is a worse fit for the ABSA literature and a much better fit for the data. The span-extraction work stays on the record as a negative result instead of being quietly deleted.

The cascade is about routing cost, not accuracy

Sentiment is labelled offline by a cascade: rules, then SenticNet, then the fine-tuned RoBERTa, then an LLM for whatever is still uncertain, with a confidence gate at each hop.

SenticNet is in there for a reason that has nothing to do with accuracy. It does not improve held-out F1 over rules alone, and I measured that carefully enough to be sure of it. In a separate 1,000-document routing probe, it cut the share needing an LLM after the rule layer from 54.8% to 49.4%. That is 5.4 percentage points less fallback, with unchanged held-out F1 in the rules-only comparison. That is the justification for keeping it. Reporting it as a modelling win would be dishonest.

An earlier serial benchmark on 30 documents measured the rule-gated hybrid at 16.42 seconds, against 116.07 seconds when every document went to the LLM. That is about a 7.1× speedup on that sample. This was the earlier rules and SenticNet pipeline, before the RoBERTa cascade. The old report’s roughly 1.9-hour versus 13.5-hour comparison concerned a 12,675-document corpus; the all-LLM time was projected from the small benchmark. Those times do not describe the later 19,187-document labeling pass.

Gating, and one idea that went in the bin

Confidence thresholds are tuned per family on a dev split rather than set once globally, because the document-level aggregation leans on them heavily and the families are not equally separable from one another.

I also tried gating on top2_gap, the margin between the top two class probabilities, on the theory that a prediction which is confident but close deserves escalating. Plain confidence thresholds turned out to be more useful, so the gap filter came back out.

The epoch comparison used the fixed stratified 70/15/15 protocol over the 1,800-document adjudicated reference corpus. The three-epoch RoBERTa subjectivity model reached 0.9257 accuracy and 0.8872 macro F1. A later seed-7 tuned subjectivity run scored 0.8808 macro F1 and did not beat it.

The 0.8610 subjectivity macro F1 in the headline comes from the four-epoch RoBERTa configuration used as the frozen end-to-end supervised benchmark. That configuration was selected under a polarity-first metric hierarchy: its polarity macro F1 was 0.6977, against 0.6926 for the three-epoch configuration. The best subjectivity score and the frozen end-to-end benchmark therefore belong to different configurations. Neither number is the F1 of the complete routed cascade or the separate aspect-family model.