Skip to content
MiraiTech
AI Lab

Experiment · LLM

Real-time sentiment stream

A fine-tuned DistilBERT model classifies each message in under 100 ms. Kafka handles ingestion at more than 10,000 messages a second, and a live dashboard shows the sentiment mix, trends and anomaly alerts.

Type
LLM
Status
Live
Models
DistilBERT (fine-tuned), RoBERTa

Results

  • 94% accuracy
  • Under 100 ms per message
  • 10k messages a second

Accuracy on the held-out test set was 94%. With 8 Kafka consumers, the pipeline sustained 10,200 messages/second. Per-message latency was 8ms at p50 and 23ms at p99. Anomaly detection took 4.7 seconds on average. The dashboard renders at 60fps with a 100ms data refresh interval.

What we tested

A sentiment analysis pipeline for streaming text that handles 10,000+ messages per second and classifies each one in under 100ms. A fine-tuned DistilBERT model served with ONNX Runtime labels messages positive, negative or neutral with 94% accuracy. A live dashboard shows the trends and flags anomalies.

Method

We fine-tuned DistilBERT on 120,000 labeled texts: 50,000 tweets (Sentiment140), 40,000 Amazon product reviews and 30,000 support tickets from an internal dataset. The model was exported to ONNX for faster inference. We compared it with RoBERTa-base, VADER and zero-shot GPT-3.5 on a held-out test set of 5,000 examples. Throughput was tested on a Kafka pipeline with simulated message streams at 1k, 5k and 10k messages/second.

Findings

  1. 01

    DistilBERT nearly matched RoBERTa at 6x the throughput

    Fine-tuned DistilBERT reached 94% accuracy against RoBERTa's 95.2%, and processed 10,200 messages/sec against RoBERTa's 1,700. For streaming, giving up 1.2% accuracy for 6x the throughput is an easy decision.

  2. 02

    ONNX Runtime doubled inference speed

    Converting the PyTorch model to ONNX and running it on ONNX Runtime cut per-message latency from 18ms to 8ms. Batching 32 messages brought amortized latency down to 2ms per message.

  3. 03

    Domain data mattered more than model size

    DistilBERT fine-tuned on support tickets scored 91% on support ticket sentiment. Zero-shot GPT-3.5 scored 78% on the same data. A small model fine-tuned on the domain beat a large general model.

  4. 04

    Sentiment shifts flagged in under 5 seconds

    A sliding-window z-score detector on the 30-second sentiment moving average raises an alert within 5 seconds of a significant shift.

How it is built

Messages arrive on Kafka topics and are consumed by a pool of classification workers running the ONNX model. Each classified message is written to ClickHouse with its timestamp, source, sentiment label and confidence score. The dashboard queries ClickHouse for live aggregations (sentiment distribution, trend lines, volume) and receives a WebSocket push when an anomaly threshold is crossed.

The fine-tuned DistilBERT model is served with ONNX Runtime. Kafka handles message ingestion and scales through consumer groups. ClickHouse stores the sentiment time series for trend analysis. FastAPI serves the classification API, and the live dashboard is built with Next.js and WebSocket updates.

  • DistilBERT
  • Kafka
  • FastAPI
  • Next.js
  • ClickHouse

Problems we hit

Sarcasm

Sarcastic messages like "Oh great, another outage" were classified as positive 40% of the time. We added a sarcasm detection head trained on 10,000 additional sarcasm-labeled examples, which brought misclassification down to 15%.

Consumer lag during bursts

Traffic spikes of 50k messages/sec caused consumer lag to build up. We set the consumer groups to scale out when lag exceeds 1,000 messages and scale back down after the burst.