LLM Fine-Tuning and Evaluation

Fine-tuning is one of the most misunderstood parts of applied AI. It sounds like the obvious next step when a general model does not behave exactly the way a business wants. In reality, fine-tuning is useful only when the problem is well-defined, the training data is strong, and the evaluation method is clear.

Many issues can be solved without fine-tuning. Better prompts, better retrieval, cleaner source data, stronger output schemas, or improved workflow design may deliver the result faster and with less risk. A custom model is worth considering when the desired behavior is repeated, specific, and hard to achieve through context alone.

Diagnose the Problem First

Before fine-tuning, identify what is actually failing.

Is the model missing company knowledge? That is usually a retrieval problem. Is it returning unstructured text when the application needs JSON? That may be an output constraint problem. Is it using the wrong tone? Prompting or examples may be enough. Is it consistently misclassifying domain-specific cases? Fine-tuning may help.

This diagnosis prevents expensive work in the wrong layer. A model cannot learn private facts that change weekly unless those facts are in training data or retrieval context. Fine-tuning can shape behavior, style, classification patterns, and domain interpretation. It is not a replacement for a knowledge base.

Build an Evaluation Set Before Training

Fine-tuning without evaluation is guesswork. The evaluation set should exist before the training run. It should include realistic examples, edge cases, and expected outputs. It should also reflect the business outcome, not just model preference.

For an invoice extraction model, evaluation might measure field-level accuracy, total reconciliation, and human correction time. For a candidate screening assistant, it might measure rubric alignment, evidence citation, and false negative rate. For a support response model, it might measure policy compliance, citation quality, tone, and escalation behavior.

The evaluation set should be separate from training data. Otherwise, the model may look improved only because it has seen the answers before.

Training Data Quality Matters More Than Volume

Fine-tuning data should show the model exactly what good behavior looks like. A few hundred excellent examples can be more valuable than thousands of inconsistent ones.

Good examples are complete, consistent, and representative. They include the input the model will see in production and the output the application expects. If the desired output is structured, the training examples should use the same schema. If the model should explain decisions, the explanations should be written in the style and level of detail you want.

Bad data teaches bad behavior. If labels are inconsistent, the model learns inconsistency. If examples include outdated policies, the model may reproduce them. If edge cases are missing, the model may fail exactly where the business needs reliability.

Dataset curation is often the real work of fine-tuning.

Compare Against Simpler Baselines

A fine-tuned model should earn its place. Compare it against the current baseline: a strong prompt, a RAG pipeline, a rules-based validator, or a larger general model.

Sometimes fine-tuning improves speed and cost by allowing a smaller model to perform a narrow task well. Sometimes it improves format consistency. Sometimes it improves domain classification. But if the gain is small, the operational cost may not be worth it.

Operational cost includes dataset maintenance, model versioning, regression testing, deployment, monitoring, and rollback plans. A custom model becomes part of the product. It needs ownership.

Fine-Tuning Does Not Remove Guardrails

A fine-tuned model can still make mistakes. It still needs validation, observability, and fallback behavior.

If the model extracts invoice totals, arithmetic validation should still check the result. If it classifies support tickets, confidence thresholds and human review should still exist. If it drafts messages, policy checks may still be necessary. Fine-tuning improves behavior, but software boundaries keep the system safe.

This is why we combine model work with clean architecture. The model provider is an adapter, not the center of the application. The domain rules stay outside the model. The use case defines what the system needs. The model is one implementation detail that can be replaced when requirements change.

Monitor After Deployment

Model quality can drift as inputs change. New suppliers, new policies, new job roles, new customer questions, and new document formats can all reduce performance. Monitoring should track real production behavior.

Useful signals include confidence distribution, validation failures, human corrections, rejected outputs, latency, cost, and user feedback. When performance drops, the team should know whether to update prompts, add retrieval sources, improve validation, or create a new training set.

Fine-tuning is not a one-time event. It is a lifecycle.

When It Is Worth It

A custom model is worth it when the task is frequent, valuable, and specific enough for training to create measurable improvement. It is especially useful when the business has high-quality examples and a clear definition of success.

It is not worth it when the problem is vague, the data is weak, or the main issue is missing knowledge. In those cases, better architecture will usually outperform model customization.

The practical path is simple: define the task, build the evaluation set, improve the baseline, curate training data, fine-tune only if the expected gain is clear, then monitor the result in production. That sequence keeps AI work tied to measurable business value instead of model enthusiasm.