Refactor Langfuse dataset upload

[fcogidi]

Summary

Refactors Langfuse dataset upload to support both JSONL and JSON inputs (with format auto-detection), adds a reusable Rich progress utility for progress UX, and adds focused unit tests for upload parsing/validation behaviour.

Clickup Ticket(s): N/A

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
• 📝 Documentation update
• 🔧 Refactoring (no functional changes)
• ⚡ Performance improvement
• 🧪 Test improvements
• 🔒 Security fix

Changes Made

• Added reusable progress helpers in aieng-eval-agents/aieng/agent_evals/progress.py (create_progress and track_with_progress) and exported them in aieng-eval-agents/aieng/agent_evals/__init__.py.
• Refactored upload_dataset_to_langfuse in aieng-eval-agents/aieng/agent_evals/langfuse.py to:
  • auto-detect JSON vs JSONL,
  • parse JSON arrays and line-by-line JSONL,
  • validate required keys (input, expected_output),
  • surface line-aware JSONL parse errors,
  • normalize metadata (id fallback handling),
  • show upload progress during dataset item creation.
• Added unit tests in aieng-eval-agents/tests/aieng/agent_evals/test_langfuse.py covering JSON upload, JSONL upload, malformed JSONL handling, missing required fields, and progress helper smoke checks.

Testing

• Tests pass locally (uv run pytest tests/)
• Type checking passes (uv run mypy <src_dir>)
• Linting passes (uv run ruff check src_dir/)
• Manual testing performed (describe below)

Manual testing details:
N/A

Screenshots/Recordings

N/A

Related Issues

N/A

Deployment Notes

Checklist

• Code follows the project's style guidelines
• Self-review of code completed
• Documentation updated (if applicable)
• No sensitive information (API keys, credentials) exposed

[Copilot]

Pull request overview

This PR refactors the Langfuse dataset upload functionality to support both JSON and JSONL formats with automatic format detection, adds reusable Rich progress utilities for improved user experience, and includes comprehensive unit tests for the new parsing and validation logic.

Changes:

• Added reusable progress bar utilities (create_progress and track_with_progress) for consistent progress visualization
• Enhanced upload_dataset_to_langfuse to auto-detect format, validate records, and provide better error messages with line numbers
• Added focused unit tests covering JSON/JSONL upload, malformed data handling, and progress helper functionality

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
aieng-eval-agents/aieng/agent_evals/progress.py	Introduces reusable Rich progress utilities with standardized column layout
aieng-eval-agents/aieng/agent_evals/langfuse.py	Refactors dataset upload with format detection, improved validation, and progress tracking
aieng-eval-agents/aieng/agent_evals/init.py	Exports new progress utilities for public API
aieng-eval-agents/tests/aieng/agent_evals/test_langfuse.py	Adds comprehensive unit tests for upload parsing, validation, and progress helpers

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[amrit110]

Just the one comment! Overall looks good.