Add page-level documentation evaluations - #4110
Draft
thodges-gh wants to merge 3 commits into
Draft
Conversation
Add schema-backed evaluation metadata and CLI inventory, change, and run workflows. Evaluate CRE Read Data Feeds guidance with distinct hosted generator and grader providers, strict output validation, and an offline known-bad fixture.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Open validated page and skill files with no-follow, nonblocking flags. Inspect and read each file through the same handle, then always close it.
Preserve no-follow, same-handle reads for CodeQL race findings. Give graders verified verbatim response candidates; exact-quote validation and two provider calls remain unchanged.
Contributor
Author
|
Output: {
"schemaVersion": 1,
"runId": "20260826T223422Z-cre-read-data-feeds-fde085a6-8c12-4c62-883c-28791c3e74c6",
"kind": "model",
"evalId": "cre-read-data-feeds",
"page": "src/content/cre-templates/read-data-feeds.mdx",
"skills": [
"chainlink-cre-skill"
],
"usableBy": [
"human",
"agent"
],
"outcomes": [
"The response recommends regular Data Feeds for single-value feeds and MVR Data Feeds for bundled multi-value feeds.",
"The response lists Go and TypeScript choices with the exact regular IDs read-data-feeds-go and read-data-feeds-ts and the exact MVR IDs read-mvr-data-feeds-go and read-mvr-data-feeds-ts.",
"The response explains that the workflow runs every 10 minutes on a cron trigger, uses the CRE chain reader to call the feed contract, and returns scaled regular-feed values or decoded MVR fields."
],
"providers": {
"generator": "openai:gpt-5.6-terra",
"grader": "openai:gpt-5.6-sol"
},
"response": "Choose based on the shape of data you need:\n\n- **Regular Data Feeds**: use these for a conventional **single-value feed**, such as BTC/USD or ETH/USD. The workflow reads `decimals()` and `latestAnswer()` from each feed contract, scales the answer using its decimals, logs it, and returns a JSON array of feed results.\n- **MVR (Multiple Variable Response) Data Feeds**: use these when one feed supplies a **bundle of multiple related values**. The workflow reads `latestBundle()` and `bundleDecimals()` from a `BundleAggregatorProxy`, decodes the returned bytes into typed fields, scales numeric fields as appropriate, and returns a structured JSON object. The provided MVR example uses the S&P Global SSA EURC feed on Base.\n\n## Template choices\n\n| Use case | Language | CRE CLI template ID |\n|---|---|---|\n| Regular single-value feeds | TypeScript | `read-data-feeds-ts` |\n| Regular single-value feeds | Go | `read-data-feeds-go` |\n| MVR bundled multi-value feeds | TypeScript | `read-mvr-data-feeds-ts` |\n| MVR bundled multi-value feeds | Go | `read-mvr-data-feeds-go` |\n\n## How the cron-based chain-reader flow works\n\n1. A **cron trigger fires every 10 minutes**.\n2. The CRE workflow uses the **CRE chain reader** to make read-only contract calls on the configured EVM network.\n3. For a regular feed, it calls:\n - `decimals()` to learn the feed’s precision.\n - `latestAnswer()` to obtain the current integer answer.\n4. The workflow applies the decimals to present the regular-feed answer as a scaled value, then returns the resulting feed entries as JSON.\n5. For an MVR feed, it calls:\n - `latestBundle()` to retrieve the encoded bundle.\n - `bundleDecimals()` to obtain precision metadata for bundle fields.\n6. It decodes the bundle bytes according to the feed’s field schema, scales numerical values, and returns the decoded fields as a structured JSON result.\n\nIn short: select **regular** templates for one price/value per feed contract; select **MVR** templates when the feed is intentionally published as a typed, multi-field bundle.",
"outcomeResults": [
{
"outcome": "The response recommends regular Data Feeds for single-value feeds and MVR Data Feeds for bundled multi-value feeds.",
"pass": true,
"evidence": "In short: select **regular** templates for one price/value per feed contract; select **MVR** templates when the feed is intentionally published as a typed, multi-field bundle.",
"reason": "The response clearly recommends each feed type according to whether the data is single-value or bundled multi-value."
},
{
"outcome": "The response lists Go and TypeScript choices with the exact regular IDs read-data-feeds-go and read-data-feeds-ts and the exact MVR IDs read-mvr-data-feeds-go and read-mvr-data-feeds-ts.",
"pass": true,
"evidence": "Choose based on the shape of data you need:\n\n- **Regular Data Feeds**: use these for a conventional **single-value feed**, such as BTC/USD or ETH/USD. The workflow reads `decimals()` and `latestAnswer()` from each feed contract, scales the answer using its decimals, logs it, and returns a JSON array of feed results.\n- **MVR (Multiple Variable Response) Data Feeds**: use these when one feed supplies a **bundle of multiple related values**. The workflow reads `latestBundle()` and `bundleDecimals()` from a `BundleAggregatorProxy`, decodes the returned bytes into typed fields, scales numeric fields as appropriate, and returns a structured JSON object. The provided MVR example uses the S&P Global SSA EURC feed on Base.\n\n## Template choices\n\n| Use case | Language | CRE CLI template ID |\n|---|---|---|\n| Regular single-value feeds | TypeScript | `read-data-feeds-ts` |\n| Regular single-value feeds | Go | `read-data-feeds-go` |\n| MVR bundled multi-value feeds | TypeScript | `read-mvr-data-feeds-ts` |\n| MVR bundled multi-value feeds | Go | `read-mvr-data-feeds-go` |\n\n## How the cron-based chain-reader flow works\n\n1. A **cron trigger fires every 10 minutes**.\n2. The CRE workflow uses the **CRE chain reader** to make read-only contract calls on the configured EVM network.\n3. For a regular feed, it calls:\n - `decimals()` to learn the feed’s precision.\n - `latestAnswer()` to obtain the current integer answer.\n4. The workflow applies the decimals to present the regular-feed answer as a scaled value, then returns the resulting feed entries as JSON.\n5. For an MVR feed, it calls:\n - `latestBundle()` to retrieve the encoded bundle.\n - `bundleDecimals()` to obtain precision metadata for bundle fields.\n6. It decodes the bundle bytes according to the feed’s field schema, scales numerical values, and returns the decoded fields as a structured JSON result.\n\nIn short: select **regular** templates for one price/value per feed contract; select **MVR** templates when the feed is intentionally published as a typed, multi-field bundle.",
"reason": "The table includes both languages and all four exact regular and MVR template IDs."
},
{
"outcome": "The response explains that the workflow runs every 10 minutes on a cron trigger, uses the CRE chain reader to call the feed contract, and returns scaled regular-feed values or decoded MVR fields.",
"pass": true,
"evidence": "Choose based on the shape of data you need:\n\n- **Regular Data Feeds**: use these for a conventional **single-value feed**, such as BTC/USD or ETH/USD. The workflow reads `decimals()` and `latestAnswer()` from each feed contract, scales the answer using its decimals, logs it, and returns a JSON array of feed results.\n- **MVR (Multiple Variable Response) Data Feeds**: use these when one feed supplies a **bundle of multiple related values**. The workflow reads `latestBundle()` and `bundleDecimals()` from a `BundleAggregatorProxy`, decodes the returned bytes into typed fields, scales numeric fields as appropriate, and returns a structured JSON object. The provided MVR example uses the S&P Global SSA EURC feed on Base.\n\n## Template choices\n\n| Use case | Language | CRE CLI template ID |\n|---|---|---|\n| Regular single-value feeds | TypeScript | `read-data-feeds-ts` |\n| Regular single-value feeds | Go | `read-data-feeds-go` |\n| MVR bundled multi-value feeds | TypeScript | `read-mvr-data-feeds-ts` |\n| MVR bundled multi-value feeds | Go | `read-mvr-data-feeds-go` |\n\n## How the cron-based chain-reader flow works\n\n1. A **cron trigger fires every 10 minutes**.\n2. The CRE workflow uses the **CRE chain reader** to make read-only contract calls on the configured EVM network.\n3. For a regular feed, it calls:\n - `decimals()` to learn the feed’s precision.\n - `latestAnswer()` to obtain the current integer answer.\n4. The workflow applies the decimals to present the regular-feed answer as a scaled value, then returns the resulting feed entries as JSON.\n5. For an MVR feed, it calls:\n - `latestBundle()` to retrieve the encoded bundle.\n - `bundleDecimals()` to obtain precision metadata for bundle fields.\n6. It decodes the bundle bytes according to the feed’s field schema, scales numerical values, and returns the decoded fields as a structured JSON result.\n\nIn short: select **regular** templates for one price/value per feed contract; select **MVR** templates when the feed is intentionally published as a typed, multi-field bundle.",
"reason": "The workflow description includes the 10-minute cron, CRE chain-reader contract calls, regular scaling, and MVR decoding."
}
],
"overallPass": true,
"providerCalls": 2
} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
inventory, skill-filtered inventory, changed-page inspection, andruncommands. Bind the first runnable evaluation to the CRE Read Data Feeds page and the registeredchainlink-cre-skill.evals/runs/.0.122.0and raise the Node.js floor to22.22.0.Verification
0.122.0; the Node.js floor is22.22.0.FAIL cre-read-data-feeds 2/3; its ignored JSON was mode0600and recordedproviderCalls: 0.Review the ignored
evals/runs/*.jsonresult.