Skip to content
MiraiTech
All work

Case study · Full stack

LLM monitoring dashboard

An observability dashboard for AI features in production. It records accuracy, latency, cost and toxicity for every model call, updates live, and is wired into CI so a release that scores below 90% on evaluations does not ship.

Year
2026
Duration
3.5 months
Type
Full stack
Status
In production
LLM monitoring dashboard, main screen

Results

  • 10k+ model calls a day
  • Dashboard refreshes every 100 ms
  • Deploys blocked below a 90% eval score

The problem

Many AI teams ship models without knowing how they behave in production. There is no standard way to track accuracy, latency and cost across thousands of inferences a day, catch quality regressions before users do, or show stakeholders that the system meets its SLAs. Teams usually find out about degradation from user complaints.

What we built

We built an observability platform for AI applications. OpenTelemetry instruments every call, DeepEval scores each inference on 15 quality metrics, and the results go into ClickHouse so dashboard queries return in under 100ms. The same evals run in CI and block a deploy when a score falls below its threshold.

Live inference monitoring
OpenTelemetry tracks every LLM call: latency (p50/p95/p99), token usage, cost per inference and error rate. The dashboard refreshes every 100ms.
15 evaluation metrics
DeepEval scores every inference on faithfulness, relevancy, hallucination, toxicity, bias, coherence and 9 more metrics. RAGAS adds RAG-specific quality scores.
Deploy gates in CI
Eval suites run as pytest fixtures in the CI pipeline. A PR that pushes any metric below its threshold is blocked, with a report of what regressed.
Anomaly alerts
Anomaly detection on rolling metric windows sends PagerDuty and Slack alerts when quality moves more than 2σ from its 7-day moving average.

How it works

Every LLM call is traced and scored. The results feed live dashboards and alerts, and the same evals gate deploys in CI.

  1. 01

    Instrumentation

    OpenTelemetry SDK wraps LLM calls → captures latency, tokens, cost and response

  2. 02

    Evaluation

    DeepEval runs 15 metrics on each inference → scores stored with the trace data

  3. 03

    Storage

    ClickHouse ingests metric events → columnar compression keeps queries on 1B+ rows under 100ms

  4. 04

    Dashboard

    Next.js + Tremor render live charts → accuracy, latency, cost and toxicity panels

  5. 05

    Alerting

    Anomaly detection on rolling windows → PagerDuty + Slack alerts on quality regressions

  6. 06

    CI/CD gate

    pytest eval suites in CI → deploy blocked if any metric drops below its threshold

Screens

LLM monitoring dashboard, screen 1
1 / 2

Decisions

ClickHouseData
We store metrics in ClickHouse rather than TimescaleDB. At 1B+ rows, its columnar compression made aggregation queries 10-50x faster, and dashboard queries that took 3s in Postgres return in 80ms.
Tremor v3.0Frontend
Its prebuilt dashboard components already follow the observability UI patterns engineers expect, so we did not need custom D3 charts. It cut frontend development time by 60%.
OpenTelemetryInfrastructure
We used OTel rather than custom instrumentation so nobody is tied to one vendor. Teams can send traces to Datadog, Grafana or Jaeger without changing application code, and the semantic conventions keep metric names consistent across services.
DeepEval 0.4.2AI / ML
Its pytest integration and ready-made metrics meant we did not have to write our own eval scripts. Eval suites are ordinary test files, so they run on the existing CI without a separate eval platform.

What went wrong

Slow dashboard queries

Problem
At 10k+ inferences a day with 15 metrics each, every chart update scanned 150k+ rows. The first version on Postgres took 3-5 seconds to load the dashboard, which is too slow for live monitoring.
Fix
We moved metric storage to ClickHouse, with materialized views for the common hourly and daily rollups. The most-viewed panels cache their query results in Redis with a 10-second TTL.
Result
Dashboard load time fell from 4.2s to 180ms. Engineers now keep the dashboard open in a tab as their monitoring view.

Eval cost in the request path

Problem
Running 15 DeepEval metrics on every inference added 800ms+ of latency and doubled LLM costs, since each metric needs extra LLM calls to make its judgment. That was not acceptable on latency-sensitive production paths.
Fix
Evaluation now runs asynchronously. Inference results return immediately, and metrics are computed in a background queue (Celery + Redis). For the CI gates, synchronous evals run on a representative 5% sample instead of every inference.
Result
No added latency on production inference. The eval queue runs with a median lag of 12 seconds, which is fast enough for the live dashboard.

Team and stack

Hiroshi designed the architecture and built the Next.js dashboard. Kenta built the OpenTelemetry instrumentation, the DeepEval pipeline and the CI gates. Hiroki built the ClickHouse storage, the rollups and the anomaly alerts.

  • Next.js 15
  • Tremor
  • ClickHouse
  • OpenTelemetry
  • DeepEval
  • RAGAS
  • PagerDuty
  • Vercel Edge