Add evaluation harness

[fcogidi]

Summary

Add evaluation harness. This includes a wrapper around Langfuse's dataset.run_experiment to run evaluators/graders on langfuse datasets, as well as a method to run trace-based evaluations.

Clickup Ticket(s): N/A

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
• 📝 Documentation update
• 🔧 Refactoring (no functional changes)
• ⚡ Performance improvement
• 🧪 Test improvements
• 🔒 Security fix

Changes Made

• Added a run_experiment, a thin wrapper around langfuse's dataset.run_experiment that includes fetching the dataset by name before running an experiment.
• Added run_trace_evaluations as a second-pass evaluation that takes the output of run_experiment, fetches traces and runs trace evaluators.
• Added run_experiment_with_trace_evals, which combines run_experiment and run_trace_evaluations in a workflow.
• Added run_coroutine_sync utility function to be able to run an async coroutine synchronously from a python script or a notebook.

Testing

• Tests pass locally (uv run pytest tests/)
• Type checking passes (uv run mypy <src_dir>)
• Linting passes (uv run ruff check src_dir/)
• Manual testing performed (describe below)

Manual testing details:
N/A

Screenshots/Recordings

N/A

Related Issues

N/A

Deployment Notes

N/A

Checklist

• Code follows the project's style guidelines
• Self-review of code completed
• Documentation updated (if applicable)
• No sensitive information (API keys, credentials) exposed

[Copilot]

Pull request overview

This PR adds an evaluation harness for running experiments and trace-based evaluations on Langfuse datasets. It provides a wrapper around Langfuse's dataset.run_experiment with additional trace evaluation capabilities.

Changes:

• Added run_experiment wrapper to fetch datasets by name and run experiments
• Added run_trace_evaluations for second-pass trace-based evaluation
• Added run_coroutine_sync utility to execute async code synchronously
• Introduced type definitions for trace evaluations, metrics, and results

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
aieng-eval-agents/aieng/agent_evals/evaluation/types.py	Defines types for evaluation harness including protocols, dataclasses, and enums
aieng-eval-agents/aieng/agent_evals/evaluation/trace.py	Implements trace-based evaluation with retry logic and metric extraction
aieng-eval-agents/aieng/agent_evals/evaluation/experiment.py	Provides experiment wrapper functions for running evaluations
aieng-eval-agents/aieng/agent_evals/evaluation/init.py	Package initialization exporting public API
aieng-eval-agents/aieng/agent_evals/async_utils.py	Added run_coroutine_sync utility and refactored progress display

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[amrit110]

Code looks great, perhaps we can adding an integration test with a real langfuse trace in a follow-up?

[amrit110]

I'd wait for @lotif to review this PR before merging, since its quite an important one.

[lotif]

Good addition, thanks. Just one question.