The Big Plan for external references, non-`$def` references, 2020-12, and much more...

[ahl]

There are a number of issues in typify and in progenitor that pertain to the handling of references. In particular, we assume that schema references will be to JSON paths of the form #/$defs/<Target> (or #/components/schemas/<Target> in OpenAPI for progenitor). This is often true! But it is by no means correct, and many OpenAPI documents and schemas contain references to other points within the file. In addition, both JSON schema and OpenAPI allow for references that target other files--also not properly handled today.

There are still more reference types! A schema may contain an $id tag that can be referenced elsewhere in the schema. This is particularly challenging because we need to take a pass through all valid schemas to discover the location of lurking reference targets. If that seems complicated, JSON Schema 2019-09 added $recursiveAnchor and $recursiveRef... which was removed and replaced in the next JSON Schema release with $dynamicAnchor and $dynamicRef. Both of these (specifically, and generically respectively) are intended for complex, multi-document, self-referential schemas. Such as those used to define the JSON schema spec itself. These last reference types ONLY seem to be used (or understood!) by the JSON Schema authors themselves, writing the JSON Schema for JSON Schema.

Much of this seems like a descent into easily avoided complexity by a standard whose guiding principles differ from my expectations... but it's what we have so let's figure it out!

In addition to these many kinds of references we don't handle today, I plan on including a couple more design goals into the proposed solution:

Multi-schema draft handling. What flavor of JSON schema does typify handle (in the most generous sense of the term)? Typify uses schemars for the Schema structure and that supports something in the vicinity of Draft 7 and 2019-09 (successive version across a naming convention change). But not really. It's missing a number of keywords from both such as contentEncoding and dependencies from Draft 7 and dependentSchemas and unevaluatedProperties from 2019-09. So, typify doesn't really handle any JSON Schema draft all that precisely or completely. Which is fine in that it works pretty well for many schemas, but it would be nice to be a little fussier when it comes to answering the question of supported schema specs.

Error reporting is lousy. Even when typify tells you what went wrong, it rarely tells you where it went wrong. If we're changing how we interact with files generally, it would be nice to keep track of where we are in files so that we can point users to the places we encounter problematic constructions.

To summarize:

• General references (i.e. not to $defs)
• External references / multi-file "bundles"
• $dynamicRef / $recursiveRef
• Strict spec conformance
• Multiple JSON Schema draft support (e.g. determine processing based on the value of $schema)
• JSON paths from errors

---

Here are some general attributes of how we intend to address this:

• Expect "bundles" of related JSON-formatted files
• Treat references more generically i.e. so that we can handle references to anywhere within the bundle
• Maintain "location" path information as we navigate schemas so that we resolve relative references (i.e. to $ids) and produce better error messages
• Serialize into version-specific structures; process into a more generic internal representation

[jrudolph]

Would be great to support split up schemas (and also directly run against a schema at a URL) because... once you get into the business of resolving non-local schemas, you get into the fun of having to resolve URLs correctly. These can be fully qualified references, absolute paths, and relative paths. And potentially referencing anywhere into those documents using fragments. Especially path resolution is something that several tools get wrong (a relative path URL reference is relative to what?).

[baxterjo]

What is the timeline for these features. Since a PR for external refs was closed in favor of this plan I would guess that this is pretty high priority?

[ahl]

@baxterjo I apologize, but I can't commit to a timeline. It's a big refactor. There are some pretty significant steps we took down the wrong path initially--each additional step adds additional work for that refactor. Neither the specific issue of external references nor this Big Plan issue are particularly high priority for Oxide so I'm working on them as time permits. If external references are a high priority for you, is maintaining a fork an option?

[baxterjo]

@ahl Thanks for the response. I decided to go down a different path for schema generation. I don't need to share my schemas with other languages or crates and have the ability to define them all myself so I went with a combination of schemars and the aide crate for generating OpenAPI specs from my rust code, so instead of this crate's function of schema -> struct I went with struct -> schema.

I'm still interested in seeing the progress of this, I've been following it for a little over a year now, when I started following it I definitely needed the schema -> struct path.

[dpezely]

When implementing "unevaluatedProperties", the W3C Music Notation Working Group has mnx-schema.json which relies upon this feature of draft 2019-09 and specifies draft 2020-12.

Integration tests which confirm as-is behavior and are ready for exercising intended behavior (by removing #[should_panic] from a test) are in a branch with further context of their use case for "unevaluatedProperties" explained in README.

(That branch is in a repo created for exercising typify's PR 912, so I'll preserve the branch for purposes here. Now that I know a bit more about non-trivial JSON schemas, I'll get back to adding tests for that PR... as time permits.)