Skip to content
MiraiTech
AI Lab

Experiment · LLM

Prompt optimization lab

Starting from a base prompt and a scoring function, it mutates and recombines prompts over several generations and keeps the best ones. A change is only accepted when an A/B test shows the improvement is statistically significant.

Type
LLM
Status
Research
Models
Claude Sonnet, GPT-4o, Gemini 2.5 Pro

Results

  • 15% average accuracy gain
  • Genetic search over prompts
  • Significance testing

Accuracy improved by 15% on average across 5 tasks, with the largest single gain (+23%) on JSON schema compliance. With regularization, the generalization gap averaged 7% (28% improvement on training, 21% on test). Each optimization run takes about 45 minutes and costs about $12 in API calls. The optimizer beat DSPy on 3 of the 5 tasks.

What we tested

Most prompts are still tuned by hand. This experiment tunes them with a genetic algorithm. Starting from a base prompt and a scoring function, it mutates, recombines and selects prompts over many generations to maximize the score on a specific evaluation suite.

Method

The genetic algorithm keeps a population of 20 prompt variants and runs 15 generations per optimization cycle. Mutations include paraphrasing, reordering instructions, swapping examples, adding or removing constraints and changing the format. Crossover combines high-scoring sections from two parent prompts. Selection is tournament selection with elitism, so the top 2 always survive. We tested on 5 tasks: classification accuracy, JSON schema compliance, code generation pass rate, summarization quality and extraction precision. Each task had a held-out test set to measure generalization.

Findings

  1. 01

    15% better than hand-written prompts on average

    Across 5 tasks, the optimizer improved accuracy by 15% on average over carefully hand-written prompts. The largest gain was in JSON schema compliance (+23%), where the optimizer found that adding negative examples cut formatting errors.

  2. 02

    Structure mattered more than wording

    The optimizer kept converging on structural patterns: instruction order, example placement and how specific the constraints are. Changes in wording mattered much less. In these tasks, prompt engineering was mostly about arranging information.

  3. 03

    Prompts overfit too

    Without a held-out test set, optimized prompts improved 28% on the training eval but only 8% on unseen examples. Prompts overfit the same way models do, so the held-out set is necessary.

  4. 04

    DSPy is a strong baseline

    Against DSPy's automatic prompt optimization, the genetic approach did about as well on simple tasks and better on complex multi-step prompts, where DSPy's optimizers struggled with larger structural changes.

How it is built

The loop creates an initial population of 20 variants from the base prompt using a range of mutations, then scores each variant with DeepEval. Parents are chosen by tournament selection, offspring come from crossover and mutation, and the offspring replace the lowest scorers. This repeats for 15 generations, and the best prompt is then validated on the held-out test set. The Streamlit dashboard shows live fitness curves, how well each mutation type works and diffs between prompts.

The genetic algorithm is written in Python with configurable mutation operators. DSPy provides the baseline comparison and some mutation primitives. DeepEval scores prompt variants on task-specific metrics. Streamlit plots fitness by generation, and Redis queues the parallel evaluation jobs.

  • Python
  • DSPy
  • DeepEval
  • Streamlit
  • Redis

Problems we hit

Evaluation cost and run time

Each run makes 15,000 LLM calls (20 variants x 50 eval examples x 15 generations). Caching, where unchanged prompt sections reuse their cached scores, and 8 parallel workers cut wall time from 6 hours to 45 minutes.

Nonsense mutations

Random mutations often produce prompts that make no sense and waste evaluation budget. We added a pre-filter: a fast model (GPT-3.5) scores each mutation for coherence before the expensive evaluation and rejects about 30% of them early.