· 8 min read
How we test and tune prompts
Structured chain-of-thought and outputs, few-shot calibration, prompt versioning, and the evaluation a prompt change goes through before production.
The limits of common prompt advice
Most prompt engineering advice online comes down to "be specific," "give examples" and "assign a role." That is fine for a ChatGPT conversation. It is not enough for a production system where a prompt runs millions of times, its output feeds downstream processes and a 2% drop in accuracy costs real money.
At MiraiTech we treat prompts as code: they are versioned, tested, reviewed and monitored. These are the practices that have made the most difference in our production systems.
Structured chain-of-thought
Adding "think step by step" is the usual way to get chain-of-thought (CoT) reasoning. It helps on benchmarks, but in production it often produces long, meandering reasoning. We use structured CoT instead, where the prompt spells out each reasoning step.
Classify the following customer support ticket.
Before answering, work through these steps:
1. IDENTIFY the product mentioned (if any)
2. DETERMINE the customer's emotional state (frustrated, neutral, satisfied)
3. CATEGORIZE the issue type from this list: [billing, technical, feature_request, account, other]
4. ASSESS urgency based on: account impact, financial impact, and time sensitivity
5. OUTPUT your classification
Ticket: """{{ticket_text}}"""
Step 1 - Product:This is how we constrain the reasoning format. When we let models reason freely, they sometimes skip steps that matter or get distracted by irrelevant details. Numbering and naming the steps gives us consistent reasoning that is easy to audit.
When CoT hurts
CoT does not always help. On simple classification tasks with clear categories, it adds latency without improving accuracy. We benchmark with and without it and keep CoT only when it gives a measurable improvement.
| Task type | CoT benefit | Notes |
|---|---|---|
| Multi-step reasoning | High (+15-25% accuracy) | Math, logic, complex analysis |
| Ambiguous classification | Medium (+5-10%) | When categories overlap |
| Simple extraction | None to negative | Adds latency, no accuracy gain |
| Creative generation | Variable | Can reduce creativity by over-constraining |
| Code generation | High (+10-20%) | Planning before coding improves correctness |
Structured outputs
Free-text LLM output is hard for downstream systems to consume. A JSON response that is valid 98% of the time means 2% of your pipeline runs crash. We enforce structure at several levels.
from pydantic import BaseModel, Field
from openai import OpenAI
from enum import Enum
class Urgency(str, Enum):
critical = "critical"
high = "high"
medium = "medium"
low = "low"
class TicketClassification(BaseModel):
"""Classification output for a customer support ticket."""
category: str = Field(
description="Issue category",
enum=["billing", "technical", "feature_request", "account", "other"]
)
urgency: Urgency
sentiment: float = Field(
ge=-1.0, le=1.0,
description="Customer sentiment from -1 (angry) to 1 (happy)"
)
requires_escalation: bool
reasoning: str = Field(
description="Brief explanation of the classification decision",
max_length=200
)
client = OpenAI()
def classify_ticket(ticket_text: str) -> TicketClassification:
completion = client.beta.chat.completions.parse(
model="gpt-4o",
messages=[
{"role": "system", "content": CLASSIFICATION_PROMPT},
{"role": "user", "content": ticket_text},
],
response_format=TicketClassification,
)
return completion.choices[0].message.parsedFew-shot calibration
Few-shot examples work, but bad examples teach bad patterns. We choose ours with a deliberate process.
The calibration process
Collect edge cases first
Do not start with easy examples. Gather the hardest, most ambiguous cases from your real data. That is where the model needs the most guidance.
Balance your example distribution
If 80% of your tickets are billing issues, do not use 4 billing examples out of 5. Over-represent rare categories so the model learns to recognize them. We usually give every category the same number of examples.
Test example ordering
Models are sensitive to example order. We shuffle our few-shot examples and measure accuracy across orderings. If accuracy varies by more than 3%, the examples are too fragile, so we add more or choose different ones.
Include negative examples
Show the model what a wrong answer looks like and explain why it is wrong. This works best for classification tasks where the category boundaries are easy to confuse.
Number of examples
For most tasks we see diminishing returns after 3-5 examples. Performance can even drop with too many, because the model starts matching surface features of the examples instead of learning the task.
Versioning prompts
Prompts change constantly in production. Without versioning, you cannot answer basic questions: what changed, when accuracy dropped, whether you can roll back.
import hashlib
from datetime import datetime
from dataclasses import dataclass
@dataclass
class PromptVersion:
name: str
template: str
version: str
created_at: datetime
metadata: dict
@property
def content_hash(self) -> str:
return hashlib.sha256(self.template.encode()).hexdigest()[:12]
class PromptRegistry:
"""Simple prompt versioning backed by a database or config file."""
def __init__(self, store):
self.store = store
def register(self, name: str, template: str, metadata: dict = None):
content_hash = hashlib.sha256(template.encode()).hexdigest()[:12]
existing = self.store.get_by_hash(name, content_hash)
if existing:
return existing # identical prompt already registered
version = f"v{self.store.next_version_number(name)}"
entry = PromptVersion(
name=name,
template=template,
version=version,
created_at=datetime.utcnow(),
metadata=metadata or {},
)
self.store.save(entry)
return entry
def get_active(self, name: str) -> PromptVersion:
return self.store.get_active(name)
def rollback(self, name: str, version: str):
self.store.set_active(name, version)Prompt changes go through code review. Each deployed prompt has a content hash, and each LLM call logs the prompt version it used. When accuracy drops, we diff the prompt versions and check them against our evaluation metrics.
Evaluation
A lot of teams ship a prompt change because "it seems better on these three examples." We run structured evaluations.
What we measure
| Dimension | How we measure | Acceptable threshold |
|---|---|---|
| Accuracy | Human-labeled golden set (200+ examples) | > 92% exact match |
| Consistency | Same input 10 times, measure variance | < 5% output variance |
| Latency | P50 and P95 response times | P95 < 3s |
| Cost | Tokens per request (input + output) | < 2000 tokens average |
| Safety | Adversarial test suite (jailbreaks, injections) | 0% bypass rate |
| Format compliance | Parse rate of structured outputs | > 99.5% |
A/B testing prompts
We A/B test prompts the same way we A/B test features. A new prompt version starts with 10% of traffic, and we monitor accuracy, latency and cost for 48 hours before ramping up. This has caught regressions that looked fine on our evaluation set but failed on real traffic.
Managing prompts in production
Template composition
A large prompt written as one block is hard to maintain, so we compose ours from sections: the system identity (who the model is and its constraints), the task specification, the required output format, few-shot calibration examples, and context injected from the application.
Each section is versioned independently and composed at runtime. That lets us update examples without touching the task specification, or tighten the output format without risking the core instructions.
Prompt injection defense
Any user-provided input is a potential injection vector. We isolate user input with delimiters (triple quotes or XML tags), validate inputs and verify outputs. No single defense is enough, so we layer them.
Cost optimization
Prompt length drives cost, so we audit our prompts for redundancy regularly. The usual wins: removing verbose instructions the model already follows by default, replacing long few-shot examples with shorter ones that make the same point, and using system-level caching for prompts that share a common prefix.
The general lesson
Written by the MiraiTech team in Tokyo
If something here is wrong, or you are working on a similar problem, write to us at contact@mirai-tech.net.