Skip to content

docs(examples): add benchSDK example benchmarks - #308

Open
dtice25 wants to merge 3 commits into
masterfrom
devin/examples
Open

docs(examples): add benchSDK example benchmarks#308
dtice25 wants to merge 3 commits into
masterfrom
devin/examples

Conversation

@dtice25

@dtice25 dtice25 commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

Rebases the examples branch onto the latest master (which added per-step concurrency/timeoutMs, the onScore hook, and --no-ingest dry-run mode) and updates the examples to cover those new primitives.

What's changed

  • Rebased devin/examples onto origin/master to pick up recent @benchsdk/runner changes.
  • Added benchmarks/examples/05-step-options.bench.ts — demonstrates ctx.step(..., { concurrency: N, timeoutMs: N }) for parallel step invocations.
  • Added benchmarks/examples/06-scoring.bench.ts — demonstrates config.onScore, lowerIsBetter, and the exported score() helper for weighted composite scoring.
  • Updated benchmarks/examples/README.md with --no-ingest / BENCHSDK_NO_INGEST instructions, the new examples, and corrected output expectations.
  • Added --no-ingest notes to the header comments of the existing examples so they can be run locally without platform credentials.

Verification

  • pnpm -r --filter "./packages/**" build succeeded.
  • pnpm typecheck passed.
  • All examples were exercised in --no-ingest mode, including the new step-options and scoring examples.
BENCHSDK_NO_INGEST=true pnpm tsx packages/benchsdk-runner/dist/bin.js run benchmarks/examples/05-step-options.bench.ts --iterations 1 --no-ingest
BENCHSDK_NO_INGEST=true pnpm tsx packages/benchsdk-runner/dist/bin.js run benchmarks/examples/06-scoring.bench.ts --iterations 5 --no-ingest

Link to Devin session: https://app.devin.ai/sessions/770c96fe577f4323ae5d7f8ca8b785dc
Requested by: @dtice25


Open in Devin Review

@dtice25 dtice25 self-assigned this Aug 12, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@open-cla

open-cla Bot commented Aug 12, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

dtice25 and others added 3 commits August 18, 2026 16:34
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ingest

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant