· 14 min read
When we fine-tune, and when we don't
How we choose between prompting, RAG and fine-tuning, and what a LoRA project involves: preparing data, training settings, evaluation and cost.
The decision teams get wrong
The most expensive LLM mistake we see is choosing the wrong technique: fine-tuning when prompt engineering would do, or prompt engineering when the task needs fine-tuning. We've shipped fine-tuned models for legal document analysis, customer support routing and medical coding, and this is how we make the call.
When fine-tuning makes sense
We fine-tune when at least two of these are true:
| Condition | Why it matters |
|---|---|
| Consistent output format required | Prompt engineering struggles with strict schema adherence across edge cases |
| Domain-specific vocabulary or reasoning | General models hallucinate domain terms or apply incorrect reasoning patterns |
| Latency budget under 500ms | Smaller fine-tuned models outperform larger prompted models |
| Cost per request must decrease | A fine-tuned 8B model is 50-100x cheaper than prompting GPT-4o |
| Thousands of labeled examples available | Fine-tuning without enough data does worse than prompting |
If none of these apply, start with prompt engineering. If only one applies, try few-shot prompting with structured output first. We've seen teams spend months fine-tuning a model for a task a good system prompt could have handled.
Data preparation
Data quality matters more than any hyperparameter choice. We spend 60-70% of total project time on data preparation.
Building the training set
import json
import hashlib
from pathlib import Path
from dataclasses import dataclass, field
from pydantic import BaseModel, validator
class TrainingExample(BaseModel):
"""A single training example in chat format."""
messages: list[dict[str, str]]
@validator("messages")
def validate_messages(cls, v):
roles = [m["role"] for m in v]
# Must start with system or user
if roles[0] not in ("system", "user"):
raise ValueError("Conversation must start with system or user message")
# Must end with assistant
if roles[-1] != "assistant":
raise ValueError("Conversation must end with assistant message")
# No empty content
for msg in v:
if not msg["content"].strip():
raise ValueError(f"Empty content in {msg['role']} message")
return v
@dataclass
class DatasetBuilder:
"""Builds and validates a fine-tuning dataset with deduplication,
quality filtering, and train/eval splitting."""
examples: list[TrainingExample] = field(default_factory=list)
seen_hashes: set[str] = field(default_factory=set)
rejected: list[dict] = field(default_factory=list)
def add_example(self, raw: dict) -> bool:
# Deduplicate by content hash
content_hash = hashlib.sha256(
json.dumps(raw, sort_keys=True).encode()
).hexdigest()[:16]
if content_hash in self.seen_hashes:
self.rejected.append({"reason": "duplicate", "hash": content_hash})
return False
try:
example = TrainingExample(**raw)
except Exception as e:
self.rejected.append({"reason": str(e), "data": raw})
return False
# Quality filters
assistant_msgs = [m for m in example.messages if m["role"] == "assistant"]
avg_response_len = sum(len(m["content"]) for m in assistant_msgs) / len(assistant_msgs)
if avg_response_len < 20:
self.rejected.append({"reason": "response_too_short", "hash": content_hash})
return False
if avg_response_len > 8000:
self.rejected.append({"reason": "response_too_long", "hash": content_hash})
return False
self.seen_hashes.add(content_hash)
self.examples.append(example)
return True
def export(self, output_dir: Path, eval_ratio: float = 0.1):
"""Export train/eval split as JSONL files."""
import random
random.seed(42)
indices = list(range(len(self.examples)))
random.shuffle(indices)
split_point = int(len(indices) * (1 - eval_ratio))
train_indices = indices[:split_point]
eval_indices = indices[split_point:]
output_dir.mkdir(parents=True, exist_ok=True)
for name, idx_set in [("train", train_indices), ("eval", eval_indices)]:
path = output_dir / f"{name}.jsonl"
with open(path, "w") as f:
for i in idx_set:
f.write(self.examples[i].json() + "\n")
print(f"{name}: {len(idx_set)} examples -> {path}")
# Write rejection log
with open(output_dir / "rejected.json", "w") as f:
json.dump(self.rejected, f, indent=2)
print(f"Rejected: {len(self.rejected)} examples")Data quality checklist
Before every fine-tuning run, we check:
- No data leakage. Eval examples must not appear in the training data. We hash-check this.
- Label consistency. Two annotators should agree on at least 90% of examples. Below that, the labels are too noisy for the model to learn a coherent pattern.
- Distribution match. Training data should roughly match the production distribution of categories. Severe imbalance teaches the model to over-predict the majority class.
- No PII. We run a PII detection pass and redact before training, because fine-tuned models can memorize and repeat their training data.
LoRA, QLoRA or full fine-tuning
| Method | VRAM required (7B model) | Training speed | Quality | Best for |
|---|---|---|---|---|
| Full fine-tune | 60+ GB | Slow | Highest | When you have budget and >50k examples |
| LoRA (r=16) | 16-24 GB | Fast | Very good | Most production use cases |
| QLoRA (4-bit) | 6-10 GB | Medium | Good | Experimentation, constrained hardware |
| Adapter fusion | 20-30 GB | Medium | Very good | Multi-task models |
We default to LoRA for production fine-tuning. For most tasks the quality gap to full fine-tuning is negligible, and LoRA needs far less hardware.
from dataclasses import dataclass
from pathlib import Path
import torch
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
TrainingArguments,
BitsAndBytesConfig,
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer
@dataclass
class FinetuneConfig:
base_model: str = "meta-llama/Llama-3.1-8B-Instruct"
dataset_path: str = "data/prepared/train.jsonl"
eval_path: str = "data/prepared/eval.jsonl"
output_dir: str = "outputs/legal-classifier-v3"
lora_r: int = 16
lora_alpha: int = 32
lora_dropout: float = 0.05
epochs: int = 3
batch_size: int = 4
gradient_accumulation: int = 8
learning_rate: float = 2e-4
max_seq_length: int = 2048
use_4bit: bool = False
def train(config: FinetuneConfig):
quantization_config = None
if config.use_4bit:
quantization_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
config.base_model,
quantization_config=quantization_config,
device_map="auto",
torch_dtype=torch.bfloat16,
attn_implementation="flash_attention_2",
)
if config.use_4bit:
model = prepare_model_for_kbit_training(model)
lora_config = LoraConfig(
r=config.lora_r,
lora_alpha=config.lora_alpha,
lora_dropout=config.lora_dropout,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
tokenizer = AutoTokenizer.from_pretrained(config.base_model)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"
training_args = TrainingArguments(
output_dir=config.output_dir,
num_train_epochs=config.epochs,
per_device_train_batch_size=config.batch_size,
gradient_accumulation_steps=config.gradient_accumulation,
learning_rate=config.learning_rate,
weight_decay=0.01,
warmup_ratio=0.03,
lr_scheduler_type="cosine",
logging_steps=10,
save_strategy="epoch",
evaluation_strategy="epoch",
bf16=True,
gradient_checkpointing=True,
report_to="wandb",
load_best_model_at_end=True,
metric_for_best_model="eval_loss",
)
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=load_jsonl(config.dataset_path),
eval_dataset=load_jsonl(config.eval_path),
tokenizer=tokenizer,
max_seq_length=config.max_seq_length,
dataset_text_field="text",
)
trainer.train()
trainer.save_model(f"{config.output_dir}/final")
return trainer.state.best_metricEvaluation
A falling training loss doesn't mean the model is good. We evaluate fine-tuned models on three axes:
Task-specific accuracy
For classification tasks we measure precision, recall and F1 per class. For generation tasks we combine automated metrics (ROUGE, BERTScore) with LLM-as-judge evaluation, where a stronger model grades the output.
Regression testing
We keep a set of "golden" examples that the base model handles correctly. If fine-tuning makes these worse, the model has overfit to the fine-tuning distribution and lost general capability. More than 5% regression on the golden set is a red flag.
Adversarial testing
We test with out-of-distribution inputs, prompt injections and edge cases. Fine-tuned models can become brittle and give confident wrong answers for inputs slightly outside the training distribution.
Deployment and cost
Serving
We serve fine-tuned models with vLLM behind a FastAPI gateway. For models under 13B parameters, a single A10G instance handles 50-100 requests/second depending on sequence length.
Cost comparison
These costs come from a legal document classification project that processes 500k documents/month (pricing as of Q4 2024):
| Approach | Cost/month | Accuracy | P95 latency |
|---|---|---|---|
| GPT-4o with 8-shot prompt | $4,200 | 93.1% | 2,400ms |
| GPT-4o-mini with 8-shot prompt | $310 | 87.4% | 890ms |
| Fine-tuned Llama 3.1 8B (LoRA) | $180 (GPU hosting) | 94.7% | 120ms |
| Fine-tuned Llama 3.1 8B (QLoRA) | $180 (GPU hosting) | 93.2% | 125ms |
For this task, the fine-tuned 8B model is cheaper, faster and more accurate than prompting GPT-4o. That's where fine-tuning pays off: narrow, well-defined tasks with enough training data.
When fine-tuning failed
We tried fine-tuning a model for open-ended legal reasoning and got worse results than prompting a frontier model. The task needed broad world knowledge and multi-step reasoning that a smaller model couldn't replicate, however good the training data was. We went back to Claude 3.5 Sonnet with a carefully built prompt chain and RAG. It costs more per request, and it gets the answers right.
The retraining loop
A fine-tuned model is never finished. Our production models go through this cycle:
- Log production inputs, outputs, confidence scores and user feedback.
- Every week, cluster low-confidence predictions and user corrections to find failure modes.
- Add the corrected examples to the training set and re-validate the distribution.
- Retrain, run the full pipeline and compare against the current production model.
- Deploy behind a feature flag and shadow test before promoting to primary.
Each cycle usually adds 1-3 percentage points of task accuracy. For most classification tasks the gains flatten out around 95-97%.
Written by the MiraiTech team in Tokyo
If something here is wrong, or you are working on a similar problem, write to us at contact@mirai-tech.net.