Skip to content
MiraiTech
All writing
Dev tools

· 9 min read

Putting evals in CI

How we test model output in CI: pipeline stages, dataset versioning, feature flags for models, cost checks, and rolling back a regression.

Where standard CI/CD falls short

We shipped our first ML feature, a document classification service, through our existing CI/CD pipeline. Unit and integration tests passed, and the model went out. Within 48 hours, classification accuracy had dropped from 94% to 71%, because production inputs had drifted from the training data distribution.

Standard CI/CD checks that the code is correct. AI applications also fail in ways code tests can't see: data quality, model performance, inference latency and cost. We needed a pipeline that treats models as first-class artifacts alongside the code.

Pipeline stages

Our pipeline for AI applications has five stages instead of the usual lint -> test -> build -> deploy:

StageWhat it verifiesBlocks deploy?
Code qualityLinting, type checks, unit testsYes
Data validationSchema conformance, distribution checksYes
Model evaluationAccuracy, latency, bias metrics against baselineYes (if regression)
Shadow deploymentSide-by-side comparison with production modelNo (advisory)
Canary releaseReal traffic on a subset of usersYes (if error rate spikes)

The evaluation pipeline

Model evaluation is the stage that matters most. Every pull request that touches model code, training data or feature engineering triggers an evaluation run:

.github/workflows/model-evaluation.yml
name: Model Evaluation
 
on:
  pull_request:
    paths:
      - 'ml/**'
      - 'data/features/**'
      - 'configs/model_*.yaml'
 
jobs:
  evaluate:
    runs-on: [self-hosted, gpu]
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v4
        with:
          lfs: true
 
      - name: Pull evaluation dataset
        run: |
          dvc pull data/eval/
          echo "Dataset version: $(dvc version)"
          echo "Eval set size: $(wc -l < data/eval/golden_set.jsonl)"
 
      - name: Run model evaluation
        run: |
          python -m ml.evaluate \
            --model-config configs/model_candidate.yaml \
            --eval-data data/eval/golden_set.jsonl \
            --baseline-metrics artifacts/baseline_metrics.json \
            --output artifacts/eval_report.json
 
      - name: Check regression thresholds
        run: |
          python -m ml.ci.check_regression \
            --report artifacts/eval_report.json \
            --max-accuracy-drop 0.02 \
            --max-latency-increase-pct 15 \
            --max-cost-increase-pct 10
 
      - name: Post evaluation summary to PR
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const report = require('./artifacts/eval_report.json');
            const body = `## Model Evaluation Results
            | Metric | Baseline | Candidate | Delta |
            |--------|----------|-----------|-------|
            | Accuracy | ${report.baseline.accuracy} | ${report.candidate.accuracy} | ${report.delta.accuracy} |
            | P95 Latency | ${report.baseline.p95_latency}ms | ${report.candidate.p95_latency}ms | ${report.delta.p95_latency}ms |
            | Cost/1k requests | $${report.baseline.cost_per_1k} | $${report.candidate.cost_per_1k} | $${report.delta.cost_per_1k} |
            `;
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body
            });

Dataset versioning

Datasets are versioned and reviewed like code, and immutable once tagged. We use DVC backed by S3 for large files. This script checks a new dataset against a reference distribution:

ml/data/validate.py
import json
import sys
from pathlib import Path
from dataclasses import dataclass
from collections import Counter
 
import numpy as np
from scipy import stats
 
 
@dataclass
class ValidationReport:
    total_samples: int
    label_distribution: dict[str, int]
    schema_violations: list[str]
    distribution_drift: float
    is_valid: bool
 
 
def validate_dataset(
    dataset_path: Path,
    reference_path: Path,
    max_drift_threshold: float = 0.1,
) -> ValidationReport:
    """Validate a dataset against a reference distribution.
 
    Uses Jensen-Shannon divergence to detect distribution drift.
    A drift score above the threshold blocks the pipeline.
    """
    with open(dataset_path) as f:
        samples = [json.loads(line) for line in f]
 
    with open(reference_path) as f:
        reference = [json.loads(line) for line in f]
 
    # Schema validation
    required_fields = {"text", "label", "source", "timestamp"}
    violations = []
    for i, sample in enumerate(samples):
        missing = required_fields - set(sample.keys())
        if missing:
            violations.append(f"Row {i}: missing fields {missing}")
        if sample.get("text") and len(sample["text"]) > 10_000:
            violations.append(f"Row {i}: text exceeds 10k chars")
 
    # Distribution drift detection
    current_dist = Counter(s["label"] for s in samples)
    ref_dist = Counter(s["label"] for s in reference)
    all_labels = sorted(set(current_dist) | set(ref_dist))
 
    current_probs = np.array([current_dist.get(l, 0) for l in all_labels], dtype=float)
    ref_probs = np.array([ref_dist.get(l, 0) for l in all_labels], dtype=float)
    current_probs /= current_probs.sum()
    ref_probs /= ref_probs.sum()
 
    drift = float(np.sqrt(stats.entropy(current_probs, ref_probs) / 2))
 
    return ValidationReport(
        total_samples=len(samples),
        label_distribution=dict(current_dist),
        schema_violations=violations,
        distribution_drift=drift,
        is_valid=len(violations) == 0 and drift < max_drift_threshold,
    )
 
 
if __name__ == "__main__":
    report = validate_dataset(
        Path(sys.argv[1]),
        Path(sys.argv[2]),
    )
    print(json.dumps(report.__dict__, indent=2))
    sys.exit(0 if report.is_valid else 1)

Feature flags for models

We deploy models behind feature flags, so we can roll back without redeploying:

ml/serving/router.py
from dataclasses import dataclass
from typing import Protocol
import hashlib
 
import launchdarkly_client as ld
 
 
class Classifier(Protocol):
    def predict(self, text: str) -> dict[str, float]: ...
 
 
@dataclass
class ModelRouter:
    """Routes inference requests to the appropriate model version
    based on feature flags and traffic allocation."""
 
    models: dict[str, Classifier]
    ld_client: ld.LDClient
 
    def route(self, user_id: str, text: str) -> dict:
        # Feature flag determines which model version to use
        model_version = self.ld_client.variation(
            "classifier-model-version",
            {"key": user_id},
            default="v2-stable",  # fallback if flag service is down
        )
 
        if model_version not in self.models:
            model_version = "v2-stable"
 
        model = self.models[model_version]
        prediction = model.predict(text)
 
        return {
            "prediction": prediction,
            "model_version": model_version,
            "routed_by": "feature_flag",
        }

A/B testing

For model A/B tests we hash the user ID so each user stays in one variant, and log every prediction with its model version for later analysis:

ComponentResponsibility
Feature flag serviceTraffic split, targeting rules, kill switch
Prediction loggerRecords model version, input hash, output, latency
Analysis pipelineComputes per-variant metrics nightly
DashboardShows A/B results with confidence intervals

Rolling back

Rolling back a model is harder than rolling back code, because model artifacts are large and take time to load. Our approach has three parts:

undefined

Keep the previous model warm

We always keep the previous model version (N-1) loaded in memory, or at least cached on disk. That lets us roll back in seconds instead of the minutes it takes to download and load a model from artifact storage.

undefined

Flip the feature flag

Changing the flag value moves all traffic to the previous model version immediately, with no deployment. This is our main rollback mechanism during production incidents.

undefined

Automated rollback triggers

We monitor the distribution of prediction confidence. If mean confidence stays more than 2 standard deviations below the baseline for 5 consecutive minutes, the system rolls back on its own and pages the on-call engineer.

Cost checks in CI

Inference costs grow with traffic and model size, so we estimate them in CI. The rates below are from early 2025; check the providers' pricing pages for current numbers:

ml/ci/cost_estimator.py
def estimate_monthly_cost(
    model_config: dict,
    estimated_monthly_requests: int,
    avg_input_tokens: int = 500,
    avg_output_tokens: int = 150,
) -> dict:
    """Estimate monthly inference cost based on model configuration."""
 
    COST_PER_1K_TOKENS = {
        "gpt-4o": {"input": 0.0025, "output": 0.01},
        "gpt-4o-mini": {"input": 0.00015, "output": 0.0006},
        "claude-3.5-sonnet": {"input": 0.003, "output": 0.015},
        "self-hosted-llama-70b": {"input": 0.0008, "output": 0.0008},
    }
 
    model_name = model_config["model"]
    costs = COST_PER_1K_TOKENS.get(model_name)
 
    if not costs:
        return {"error": f"Unknown model: {model_name}", "estimated_cost": None}
 
    monthly_input_cost = (estimated_monthly_requests * avg_input_tokens / 1000) * costs["input"]
    monthly_output_cost = (estimated_monthly_requests * avg_output_tokens / 1000) * costs["output"]
    total = monthly_input_cost + monthly_output_cost
 
    return {
        "model": model_name,
        "monthly_requests": estimated_monthly_requests,
        "monthly_input_cost": round(monthly_input_cost, 2),
        "monthly_output_cost": round(monthly_output_cost, 2),
        "total_monthly_cost": round(total, 2),
        "cost_per_request": round(total / estimated_monthly_requests, 6),
    }

CI posts the cost delta as a comment on every PR. If switching from gpt-4o-mini to claude-3.5-sonnet would add $3,000 to the monthly bill, the team knows before merging instead of finding out from the invoice.

What we got wrong at first

Evaluating only on static benchmarks

Our first eval suite used a frozen test set that was clean and balanced. Production data was messy, imbalanced and full of edge cases. We now keep two eval sets: a clean benchmark for tracking regressions, and a "production-like" set sampled from real user inputs with PII stripped.

Treating model updates like code updates

Early on, we deployed model updates with the same confidence as code changes. But a model change can degrade in ways that take days to surface: accuracy can hold on common cases while it collapses on tail queries. Shadow deployments, where the new model runs alongside production without serving results, are how we catch these slow regressions now.

Written by the MiraiTech team in Tokyo

If something here is wrong, or you are working on a similar problem, write to us at contact@mirai-tech.net.