Evaluating Fine-Tuned Models

Why Evaluation Matters

Without proper evaluation, you can't know if fine-tuning actually improved your model. Worse, you might deploy a model that's regressed on important capabilities (catastrophic forgetting).

Evaluation Approaches

1. Automated Metrics

• Perplexity: How surprised is the model by the test data? Lower is better, but doesn't directly measure quality.
• BLEU/ROUGE: N-gram overlap with reference texts. Useful for translation and summarization.
• Pass@k: For code generation — does the generated code pass test cases?
• Exact match: For structured outputs (JSON, classification labels).

2. LLM-as-Judge

Use a strong model (Claude Opus, GPT-4) to evaluate the outputs of your fine-tuned model:

Rate the following response on a scale of 1-5 for:
- Accuracy: Is the information correct?
- Relevance: Does it answer the question?
- Completeness: Does it cover all important aspects?
- Style: Does it match the desired tone/format?

Question: {question}
Response: {model_output}

3. Human Evaluation

The gold standard. Have domain experts rate outputs on relevant criteria. Expensive but necessary for high-stakes applications.

Evaluation Best Practices

• Held-out test set: Never evaluate on training data
• Compare to baseline: Always compare fine-tuned vs. base model
• Test for regression: Evaluate on general benchmarks to detect capability loss
• A/B testing: In production, compare new model against the current one with real users
• Diverse test cases: Include easy, medium, and hard examples plus edge cases

🌼 Daisy+ in Action: Real-World Evaluation

Daisy+ evaluates its AI agents through real-world metrics: customer satisfaction ratings on livechat conversations, email response accuracy (did the AI correctly categorize and respond?), and task completion rates for automated workflows. These aren't academic benchmarks — they're business KPIs that directly measure whether the AI is helping or hindering.