Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 20 additions & 3 deletions nodejs/docs/factories.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,13 @@ const reviewChanged = defineFactory({
"Review changed files and verify the findings. " +
"args: { files: string[] } — the paths to review.",
phases: [{ title: "Review" }, { title: "Verify" }],
argsSchema: {
type: "object",
required: ["files"],
properties: {
files: { type: "array", items: { type: "string" } },
},
},
limits: {
maxConcurrentSubagents: 3,
maxTotalSubagents: 10,
Expand All @@ -41,9 +48,19 @@ const reviewChanged = defineFactory({
const session = await joinSession({ factories: [reviewChanged] });
```

Factory metadata contains a stable `name`, a human-readable `description`, declared `phases`, and optional `limits`. Phase entries contain a `title` and optional `detail`.
Factory metadata contains a stable `name`, a human-readable `description`, declared `phases`, an optional `argsSchema`, and optional `limits`. Phase entries contain a `title` and optional `detail`.

## Declaring an argument shape

A factory that reads `ctx.args` should declare `meta.argsSchema`, as the example above does. When the model invokes the factory through the `run_factory` tool, the CLI validates `args` against the declaration **before** the run starts.

Declaring one turns an expensive failure into a cheap one. With a schema, a malformed call is rejected up front — the model gets a correction hint and retries, and no run row, permission prompt, or credit spend happens. Without one, nothing validates: the run starts, takes a user approval, spends credits, and then dies inside the factory body with a confusing error. Agents can read the declared shape with `factories_manage` using `operation: "inspect"`.

Enforcement covers structure — types, required properties, and enum or const values. Finer constraints such as `minLength`, `pattern`, or `additionalProperties` are recorded in the declaration but not enforced. The accepted vocabulary is the `FactoryJsonSchema` subset also used for subagent structured output: `type`, `required`, `enum`, `const`, recursive `properties`/`items`, and `anyOf`/`oneOf`/`allOf`. A `type` is one of `null`, `boolean`, `integer`, `number`, `string`, `array`, or `object`, or a non-empty array of those such as `["object", "null"]`. A declaration outside that subset is rejected at registration.

`argsSchema` is optional and backward compatible. A factory that omits it behaves exactly as before, so **the `description` is then the only thing telling an agent what arguments to supply** — state the expected shape there.

There is no declared schema for `ctx.args`. The `run_factory` tool forwards `args` verbatim and its parameter is untyped, so **the `description` is the only thing telling an agent what arguments to supply** — state the expected shape there whenever a factory reads `ctx.args`, as the example above does. Arguments supplied by an extension calling `session.factory.run(...)` directly are typed through `defineFactory<TArgs>`, but that typing does not reach the model. A factory that reads `ctx.args` should validate it rather than assume a shape.
Validation covers the model's `run_factory` path only. An extension calling `session.factory.run(...)` directly is not validated against `argsSchema`; those arguments are typed through `defineFactory<TArgs>` instead, and that typing does not reach the model. So a factory that reads `ctx.args` should still validate it rather than assume a shape — the declared subset does not enforce every constraint, and it does not run at all on the SDK path.

`defineFactory<TArgs, TResult>` accepts a `run(context)` function returning `Promise<TResult>`, where `TResult` is `JsonValue | void`. Objects, arrays, strings, numbers, booleans, and `null` are valid results. Returning `undefined` completes the factory with no result. Other non-JSON values are rejected.

Expand Down Expand Up @@ -193,7 +210,7 @@ async ({ args, agent, phase }) => {
};
```

Authoring registers the factory but does not run it. Invoke it afterwards with `run_factory`. Use `factories_manage` with `operation: "list"` to see the factories already registered in the session and `operation: "inspect"` to read one factory's description, phases, and limits before running it.
Authoring registers the factory but does not run it. Invoke it afterwards with `run_factory`. Use `factories_manage` with `operation: "list"` to see the factories already registered in the session and `operation: "inspect"` to read one factory's description, phases, declared argument shape, and limits before running it.

## Observe a run

Expand Down
12 changes: 8 additions & 4 deletions nodejs/src/factory.ts
Original file line number Diff line number Diff line change
Expand Up @@ -62,12 +62,16 @@ export type JsonValue =
| { [key: string]: JsonValue };

/**
* Conservative JSON shape language accepted for structured factory agent output.
* Conservative JSON shape language accepted by the Agent Factories surface, for
* both structured factory agent output and a factory's declared `argsSchema`.
*
* This is a best-effort structural guard used to decide whether a subagent's
* structured output should be accepted or retried — **not** a full JSON Schema
* This is a best-effort structural guard — used to decide whether a subagent's
* structured output should be accepted or retried, and whether a caller's
* factory `args` match the declared shape — **not** a full JSON Schema
* validator. Only these keywords are honored: `type`, `required`, `enum`,
* `const`, recursive `properties`/`items`, and `anyOf`/`oneOf`/`allOf`.
* `const`, recursive `properties`/`items`, and `anyOf`/`oneOf`/`allOf`. A `type`
* is one of `null`, `boolean`, `integer`, `number`, `string`, `array`, or
* `object`, or a non-empty array of those (for example `["object", "null"]`).
*
* Everything else is **ignored, not enforced**. In particular, string
* constraints (`pattern`, `minLength`, `maxLength`, `format`), numeric ranges
Expand Down
26 changes: 25 additions & 1 deletion nodejs/src/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ import type {
SessionEvent as GeneratedSessionEvent,
} from "./generated/session-events.js";
import type { CopilotSession } from "./session.js";
import type { JsonValue } from "./factory.js";
import type { FactoryJsonSchema, JsonValue } from "./factory.js";
import type {
GitHubTelemetryNotification,
ModelBillingTokenPrices,
Expand Down Expand Up @@ -2007,6 +2007,30 @@ export interface FactoryMeta {
description: string;
/** Display metadata for the progress phases the factory may report. */
phases: Array<{ title: string; detail?: string }>;
/**
* Optional declared shape of the arguments this factory expects as `ctx.args`.
*
* Declaring one is strongly recommended for any factory that reads `ctx.args`.
* When the model invokes the factory through the `run_factory` tool, the CLI
* validates `args` against this declaration **before** the run starts, so a
* malformed call is rejected with a correction hint and retried without ever
* creating a run row, prompting the user for permission, or spending credits. A
* factory that declares nothing is never validated: a malformed call starts,
* takes an approval, spends credits, and then fails inside the factory body.
* `factories_manage` with `operation: "inspect"` reports the declared shape so an
* agent can read it before invoking.
*
* This covers the model's `run_factory` path only. `session.factory.run(...)` is
* not validated against the declaration, so a factory should still check
* `ctx.args` rather than assume the declared shape held.
*
* Enforcement covers structure — types, required properties, and enum/const
* values. Finer constraints such as `minLength`, `pattern`, and
* `additionalProperties` are recorded in the declaration but not enforced. See
* {@link FactoryJsonSchema} for the accepted subset. A declaration outside that
* subset is rejected at registration.
*/
argsSchema?: FactoryJsonSchema;
/** Optional resource ceilings presented to the user before execution. */
limits?: FactoryLimits;
}
Expand Down
9 changes: 9 additions & 0 deletions nodejs/test/e2e/fixtures/factory-extension.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,15 @@ const argumentEcho = defineFactory({
name: "argument-echo",
description: "Return the invocation arguments verbatim.",
phases: [],
// Proves a declared shape survives the SDK boundary and registers against a
// real runtime. It does not exercise enforcement: `argsSchema` is checked by
// the model's `run_factory` tool, and these tests invoke `session.factory.run`,
// which does not validate. The declaration stays as wide as this factory's
// actual contract — it echoes any JsonValue, and is called with an array, an
// object, and nothing — so it cannot constrain the runs below.
argsSchema: {
type: ["object", "array", "string", "number", "integer", "boolean", "null"],
},
},
run: async ({ args }) => args,
});
Expand Down
99 changes: 99 additions & 0 deletions nodejs/test/factory.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import {
type FactoryAgentOptions,
type FactoryContext,
type FactoryDefinition,
type FactoryJsonSchema,
type JsonValue,
} from "../src/factory.js";

Expand Down Expand Up @@ -461,6 +462,104 @@ describe("factories", () => {
);
});

it("carries a declared argsSchema through defineFactory into the registration payload", async () => {
const client = new CopilotClient();
await client.start();
onTestFinished(() => stopClient(client));

const argsSchema = {
type: "object",
required: ["repoPath"],
properties: {
repoPath: { type: "string" },
depth: { type: ["integer", "null"] },
mode: { enum: ["fast", "thorough"] },
},
} satisfies FactoryJsonSchema;
const meta = {
name: "declares-args",
description: "Declares the argument shape it expects",
phases: [],
argsSchema,
};
const factory = defineFactory({ meta, run: async () => ({ ok: true }) });

// The declaration is snapshotted and deep-frozen like the rest of the
// metadata, so it cannot be mutated after registration.
expect(factory.meta.argsSchema).toEqual(argsSchema);
expect(factory.meta.argsSchema).not.toBe(argsSchema);
expect(Object.isFrozen(factory.meta.argsSchema)).toBe(true);
expect(() => {
// @ts-expect-error handle.meta.argsSchema is deeply readonly.
factory.meta.argsSchema!.type = "array";
}).toThrow(TypeError);

const omitted = defineFactory({
meta: { name: "omits-args", description: "Declares nothing", phases: [] },
run: async () => ({ ok: true }),
});
expect(omitted.meta.argsSchema).toBeUndefined();
expect("argsSchema" in omitted.meta).toBe(false);

const sendRequest = vi
.spyOn(
(client as never as { connection: { sendRequest: Function } }).connection,
"sendRequest"
)
.mockImplementation(async (method: string, params: Record<string, unknown>) => {
if (method === "session.resume") {
return { sessionId: params.sessionId };
}
throw new Error(`Unexpected method: ${method}`);
});

await client.resumeSessionForExtension(
"session-args-schema",
{ onPermissionRequest: () => ({ kind: "approved" }) },
[factory, omitted]
);

const payload = sendRequest.mock.calls.find(
([method]) => method === "session.resume"
)![1] as { factories: Array<Record<string, unknown>> };
// The schema has to survive JSON serialization to reach the runtime, which
// validates `args` against it before a run row exists.
expect(JSON.parse(JSON.stringify(payload.factories))[0].argsSchema).toEqual(argsSchema);
expect(payload.factories[1]).not.toHaveProperty("argsSchema");
});

it("documents argsSchema consistently with the runtime's enforced subset", () => {
const publicTypes = readFileSync(new URL("../src/types.ts", import.meta.url), "utf8");
const publicApi = readFileSync(new URL("../src/factory.ts", import.meta.url), "utf8");
const guide = readFileSync(new URL("../docs/factories.md", import.meta.url), "utf8");
const normalizeJSDoc = (document: string) =>
document.replace(/\r?\n\s*\* ?/g, " ").replace(/\s+/g, " ");

expect(publicTypes).toContain("argsSchema?: FactoryJsonSchema;");

// The `run_factory` tool tells the model exactly this. The two surfaces
// have to agree about what a declaration does and does not enforce.
for (const document of [normalizeJSDoc(publicTypes), guide]) {
expect(document).toContain("types, required properties, and enum");
expect(document).toMatch(
/`minLength`, `pattern`,? (?:and|or) `additionalProperties` are recorded/
);
}
expect(normalizeJSDoc(publicTypes)).toContain("before** the run starts");
// Enforcement is tool-path only: `toolRunFactoryValidateArgs` is called from
// the runtime's runFactoryTool, and never from `session.factory.run`. Both
// surfaces must keep saying so, or authors will assume their own SDK-initiated
// runs are checked.
expect(normalizeJSDoc(publicTypes)).toContain(
"`session.factory.run(...)` is not validated against the declaration"
);
expect(guide).toContain("Validation covers the model's `run_factory` path only");
expect(normalizeJSDoc(publicApi)).toContain(
"`null`, `boolean`, `integer`, `number`, `string`, `array`, or `object`"
);
expect(guide).toContain("no run row, permission prompt, or credit spend happens");
});

it("serializes only factory metadata in the extension resume payload", async () => {
const client = new CopilotClient();
await client.start();
Expand Down
Loading