Skip to content

Add page-level documentation evaluations - #4110

Draft
thodges-gh wants to merge 3 commits into
mainfrom
docs/issue-29-page-evaluation
Draft

Add page-level documentation evaluations#4110
thodges-gh wants to merge 3 commits into
mainfrom
docs/issue-29-page-evaluation

Conversation

@thodges-gh

Copy link
Copy Markdown
Contributor

Description

  • Add strict, schema-backed page evaluation metadata and enroll three documentation pages for human and agent use.
  • Add inventory, skill-filtered inventory, changed-page inspection, and run commands. Bind the first runnable evaluation to the CRE Read Data Feeds page and the registered chainlink-cre-skill.
  • Generate and grade one response against the page's ordered outcomes, or run the offline known-bad fixture with zero provider calls. Store local results under ignored evals/runs/.
  • Pin Promptfoo to 0.122.0 and raise the Node.js floor to 22.22.0.

Verification

  • Promptfoo resolved and is pinned to 0.122.0; the Node.js floor is 22.22.0.
  • Full Jest: 11 suites and 244 tests passed, exit 0.
  • Typecheck exited 0.
  • Full lint exited 0 with 0 errors and 111 existing warnings.
  • Format check exited 0.
  • Focused page evaluator: 86 tests passed, exit 0.
  • Inventory, skill-filtered inventory, and changed-page demos each exited 0.
  • Final known-bad smoke exited 0 and reported FAIL cre-read-data-feeds 2/3; its ignored JSON was mode 0600 and recorded providerCalls: 0.
  • Independent parser, output, and path reviews passed with no high- or medium-severity findings remaining.
npm run page-evaluation -- run --eval cre-read-data-feeds --skills-repo ../chainlink-agent-skills

Review the ignored evals/runs/*.json result.

Add schema-backed evaluation metadata and CLI inventory, change, and run
workflows. Evaluate CRE Read Data Feeds guidance with distinct hosted
generator and grader providers, strict output validation, and an offline
known-bad fixture.
@vercel

vercel Bot commented Aug 26, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
documentation Ready Ready Preview Aug 26, 2026 10:45pm

Request Review

Comment thread src/scripts/page-evaluation.ts Fixed
Comment thread src/scripts/page-evaluation.ts Fixed
Open validated page and skill files with no-follow, nonblocking flags.
Inspect and read each file through the same handle, then always close it.
Preserve no-follow, same-handle reads for CodeQL race findings.

Give graders verified verbatim response candidates; exact-quote validation and two provider calls remain unchanged.
@thodges-gh

Copy link
Copy Markdown
Contributor Author

Output:

{
  "schemaVersion": 1,
  "runId": "20260826T223422Z-cre-read-data-feeds-fde085a6-8c12-4c62-883c-28791c3e74c6",
  "kind": "model",
  "evalId": "cre-read-data-feeds",
  "page": "src/content/cre-templates/read-data-feeds.mdx",
  "skills": [
    "chainlink-cre-skill"
  ],
  "usableBy": [
    "human",
    "agent"
  ],
  "outcomes": [
    "The response recommends regular Data Feeds for single-value feeds and MVR Data Feeds for bundled multi-value feeds.",
    "The response lists Go and TypeScript choices with the exact regular IDs read-data-feeds-go and read-data-feeds-ts and the exact MVR IDs read-mvr-data-feeds-go and read-mvr-data-feeds-ts.",
    "The response explains that the workflow runs every 10 minutes on a cron trigger, uses the CRE chain reader to call the feed contract, and returns scaled regular-feed values or decoded MVR fields."
  ],
  "providers": {
    "generator": "openai:gpt-5.6-terra",
    "grader": "openai:gpt-5.6-sol"
  },
  "response": "Choose based on the shape of data you need:\n\n- **Regular Data Feeds**: use these for a conventional **single-value feed**, such as BTC/USD or ETH/USD. The workflow reads `decimals()` and `latestAnswer()` from each feed contract, scales the answer using its decimals, logs it, and returns a JSON array of feed results.\n- **MVR (Multiple Variable Response) Data Feeds**: use these when one feed supplies a **bundle of multiple related values**. The workflow reads `latestBundle()` and `bundleDecimals()` from a `BundleAggregatorProxy`, decodes the returned bytes into typed fields, scales numeric fields as appropriate, and returns a structured JSON object. The provided MVR example uses the S&P Global SSA EURC feed on Base.\n\n## Template choices\n\n| Use case | Language | CRE CLI template ID |\n|---|---|---|\n| Regular single-value feeds | TypeScript | `read-data-feeds-ts` |\n| Regular single-value feeds | Go | `read-data-feeds-go` |\n| MVR bundled multi-value feeds | TypeScript | `read-mvr-data-feeds-ts` |\n| MVR bundled multi-value feeds | Go | `read-mvr-data-feeds-go` |\n\n## How the cron-based chain-reader flow works\n\n1. A **cron trigger fires every 10 minutes**.\n2. The CRE workflow uses the **CRE chain reader** to make read-only contract calls on the configured EVM network.\n3. For a regular feed, it calls:\n   - `decimals()` to learn the feed’s precision.\n   - `latestAnswer()` to obtain the current integer answer.\n4. The workflow applies the decimals to present the regular-feed answer as a scaled value, then returns the resulting feed entries as JSON.\n5. For an MVR feed, it calls:\n   - `latestBundle()` to retrieve the encoded bundle.\n   - `bundleDecimals()` to obtain precision metadata for bundle fields.\n6. It decodes the bundle bytes according to the feed’s field schema, scales numerical values, and returns the decoded fields as a structured JSON result.\n\nIn short: select **regular** templates for one price/value per feed contract; select **MVR** templates when the feed is intentionally published as a typed, multi-field bundle.",
  "outcomeResults": [
    {
      "outcome": "The response recommends regular Data Feeds for single-value feeds and MVR Data Feeds for bundled multi-value feeds.",
      "pass": true,
      "evidence": "In short: select **regular** templates for one price/value per feed contract; select **MVR** templates when the feed is intentionally published as a typed, multi-field bundle.",
      "reason": "The response clearly recommends each feed type according to whether the data is single-value or bundled multi-value."
    },
    {
      "outcome": "The response lists Go and TypeScript choices with the exact regular IDs read-data-feeds-go and read-data-feeds-ts and the exact MVR IDs read-mvr-data-feeds-go and read-mvr-data-feeds-ts.",
      "pass": true,
      "evidence": "Choose based on the shape of data you need:\n\n- **Regular Data Feeds**: use these for a conventional **single-value feed**, such as BTC/USD or ETH/USD. The workflow reads `decimals()` and `latestAnswer()` from each feed contract, scales the answer using its decimals, logs it, and returns a JSON array of feed results.\n- **MVR (Multiple Variable Response) Data Feeds**: use these when one feed supplies a **bundle of multiple related values**. The workflow reads `latestBundle()` and `bundleDecimals()` from a `BundleAggregatorProxy`, decodes the returned bytes into typed fields, scales numeric fields as appropriate, and returns a structured JSON object. The provided MVR example uses the S&P Global SSA EURC feed on Base.\n\n## Template choices\n\n| Use case | Language | CRE CLI template ID |\n|---|---|---|\n| Regular single-value feeds | TypeScript | `read-data-feeds-ts` |\n| Regular single-value feeds | Go | `read-data-feeds-go` |\n| MVR bundled multi-value feeds | TypeScript | `read-mvr-data-feeds-ts` |\n| MVR bundled multi-value feeds | Go | `read-mvr-data-feeds-go` |\n\n## How the cron-based chain-reader flow works\n\n1. A **cron trigger fires every 10 minutes**.\n2. The CRE workflow uses the **CRE chain reader** to make read-only contract calls on the configured EVM network.\n3. For a regular feed, it calls:\n   - `decimals()` to learn the feed’s precision.\n   - `latestAnswer()` to obtain the current integer answer.\n4. The workflow applies the decimals to present the regular-feed answer as a scaled value, then returns the resulting feed entries as JSON.\n5. For an MVR feed, it calls:\n   - `latestBundle()` to retrieve the encoded bundle.\n   - `bundleDecimals()` to obtain precision metadata for bundle fields.\n6. It decodes the bundle bytes according to the feed’s field schema, scales numerical values, and returns the decoded fields as a structured JSON result.\n\nIn short: select **regular** templates for one price/value per feed contract; select **MVR** templates when the feed is intentionally published as a typed, multi-field bundle.",
      "reason": "The table includes both languages and all four exact regular and MVR template IDs."
    },
    {
      "outcome": "The response explains that the workflow runs every 10 minutes on a cron trigger, uses the CRE chain reader to call the feed contract, and returns scaled regular-feed values or decoded MVR fields.",
      "pass": true,
      "evidence": "Choose based on the shape of data you need:\n\n- **Regular Data Feeds**: use these for a conventional **single-value feed**, such as BTC/USD or ETH/USD. The workflow reads `decimals()` and `latestAnswer()` from each feed contract, scales the answer using its decimals, logs it, and returns a JSON array of feed results.\n- **MVR (Multiple Variable Response) Data Feeds**: use these when one feed supplies a **bundle of multiple related values**. The workflow reads `latestBundle()` and `bundleDecimals()` from a `BundleAggregatorProxy`, decodes the returned bytes into typed fields, scales numeric fields as appropriate, and returns a structured JSON object. The provided MVR example uses the S&P Global SSA EURC feed on Base.\n\n## Template choices\n\n| Use case | Language | CRE CLI template ID |\n|---|---|---|\n| Regular single-value feeds | TypeScript | `read-data-feeds-ts` |\n| Regular single-value feeds | Go | `read-data-feeds-go` |\n| MVR bundled multi-value feeds | TypeScript | `read-mvr-data-feeds-ts` |\n| MVR bundled multi-value feeds | Go | `read-mvr-data-feeds-go` |\n\n## How the cron-based chain-reader flow works\n\n1. A **cron trigger fires every 10 minutes**.\n2. The CRE workflow uses the **CRE chain reader** to make read-only contract calls on the configured EVM network.\n3. For a regular feed, it calls:\n   - `decimals()` to learn the feed’s precision.\n   - `latestAnswer()` to obtain the current integer answer.\n4. The workflow applies the decimals to present the regular-feed answer as a scaled value, then returns the resulting feed entries as JSON.\n5. For an MVR feed, it calls:\n   - `latestBundle()` to retrieve the encoded bundle.\n   - `bundleDecimals()` to obtain precision metadata for bundle fields.\n6. It decodes the bundle bytes according to the feed’s field schema, scales numerical values, and returns the decoded fields as a structured JSON result.\n\nIn short: select **regular** templates for one price/value per feed contract; select **MVR** templates when the feed is intentionally published as a typed, multi-field bundle.",
      "reason": "The workflow description includes the 10-minute cron, CRE chain-reader contract calls, regular scaling, and MVR decoding."
    }
  ],
  "overallPass": true,
  "providerCalls": 2
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants