docs: add OrcaReplay to Observability (#7365) - #7366
Open
xizhuomengcontin wants to merge 3 commits into
Open
Conversation
OrcaReplay is an open-source record/replay debugger for agents. It
records a run to a file, then replays that file with the model served
from the recording — so a run reproduces exactly, offline, without
spending tokens, and can be forked from any step onto a different model.
Unlike the other tools in this section it needs no code change: no SDK,
no callback, no import. `orca record generic-openai -- python your_crew.py`
runs the crew you already have.
Everything on the page was measured against crewai 1.15.20 before
submitting, including the console output, which is copied from a real
run rather than typed. Two CrewAI-specific notes are included because
they are what a reader will hit:
- the native provider (1.x moved LiteLLM to an optional extra) reads
both OPENAI_API_BASE and OPENAI_BASE_URL
- a bare model name always reaches that provider, while a prefixed one
is checked against a known-model list first — which matters when the
model name is a gateway's rather than the vendor's
CrewAI also has its own check in OrcaReplay's CI: a real Agent, Task and
Crew recorded against a stub origin, the origin stopped, the recording
replayed offline. So "CrewAI works with OrcaReplay" fails a build rather
than ageing quietly.
English only. The nav entry is under edge/en, so the docs freeze picks
it up for the next version per crewAIInc#6349.
The table said a prefixed name no native provider claims always raises. It resolves wherever LiteLLM is installed — CrewAI falls back to it, and 1.x simply does not install it by default. The machine this was first measured on had LiteLLM present but broken, which crewai.llm cannot tell from absent. Corrected before review rather than after. The recommendation is unchanged: use the bare form for a gateway's own model names.
CONTRIBUTING.md asks for translation parity, and crewAIInc#7311 shows it is enforced in practice rather than aspirational — it landed in all four locales. Code blocks, URLs, heading structure and table shape are identical across the four; only prose and code comments are translated, which matches what the existing ko and pt-BR pages do. Nav entries and the overview Card are added per locale, each pointing at its own.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (9)
Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review. 📝 WalkthroughWalkthroughChangesOrcaReplay observability documentation
Suggested reviewers: Merge Risk: ⚪ Minimal · up to This adds OrcaReplay observability guidance and navigation in English, Arabic, Korean, and Brazilian Portuguese. The documented pages and localized routes are aligned, with no remaining merge-blocking risk. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Warning Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #7365.
Replaces #7364, which I opened before reading the contributing guide properly and which the bot closed for having no linked issue. My mistake: I looked for a contributing guide in the repository root and missed that it lives under
.github/.What this adds
A page for OrcaReplay under Observability, in all four locales.
OrcaReplay is an open-source (Apache-2.0) record/replay debugger for agents. It records a run to a file, then replays that file with the model served from the recording — so a run reproduces exactly, offline, without spending tokens, and can be forked from any step onto a different model.
It differs from the tools currently in that section in one way worth stating plainly: there is nothing to add to your crew. No SDK, no callback handler, no import.
Files
docs/edge/{en,ar,ko,pt-BR}/observability/orcareplay.mdxdocs/edge/{en,ar,ko,pt-BR}/observability/overview.mdx<Card>eachdocs/docs.jsonedge/edgeonly, so the docs freeze carries it to the next version through the Edge nav, per #6349. Translation parity because CONTRIBUTING asks for it and #7311 shows it is how docs actually land here.Code blocks, URLs, heading structure and table shape are identical across the four; only prose and code comments are translated, matching what the existing
koandpt-BRpages do.Everything on the page was measured, and one claim was corrected before submitting
Against crewai 1.15.20. The console blocks are copied from a real run rather than typed. Three CrewAI-specific notes are included because they are what a reader will actually hit:
crewai.llms.providers.openai.completion, which reads bothOPENAI_API_BASEandOPENAI_BASE_URL— which is why the command needs no arguments.LLM(model="openai/my-gateway-model")resolves where LiteLLM is installed and raisesImportErrorwhere it is not. This matters when the model name is a gateway's rather than the vendor's.The second of those was wrong in an earlier draft — I had measured it on a machine whose LiteLLM install was broken, which
crewai.llm._ensure_litellm()cannot distinguish from an absent one, and wrote the result down as CrewAI's behaviour. Our CI, where LiteLLM works, disagreed. It is now stated conditionally and pinned by a script that checks it against whichever way LiteLLM is on the machine running it.Verified in CI, on our side
CrewAI has its own check in OrcaReplay's integration matrix — a real
Agent,TaskandCrewrecorded against a stub origin, the origin stopped, and the recording replayed offline:It runs on every commit, so the claim this page makes fails a build rather than ageing quietly. The check is
crewai_agent.py.Happy to adjust tone, length, placement or the translations to whatever you would rather have.