Suggestion: Fix the evaluation mode for a specific subtask beforehand

[greentfrapp]

Hi, thanks for creating ResearchClawBench! I'm considering doing a submission soon.

As I was reviewing the runs from the public leaderboard, I noticed that the same sub-task may be evaluated as Mode A (Objective Evaluation) in some cases and Mode B (Subjective Evaluation) in other cases. This means the judge may be applying inconsistent scoring rubrics.

For example, for the Physics 000 task, if we compare the public ResearchHarness runs for Claude Opus 4.6 and 4.7, we see:

• For ResearchHarness (Claude-Opus-4.6): the judge determines Result 1 as Mode B, Result 2 as Mode A, and Result 3 as Mode A.
• But for ResearchHarness (Claude-Opus-4.7): the judge determines Result 1 as Mode A, Result 2 as Mode A, and Result 3 as Mode B.
  Specifically, Results 1 and 3 are evaluated with different modes across the two runs.

Would you consider fixing the evaluation mode for each subtask beforehand? This could eliminate an additional source of noise in the evaluation.

[black-yt]

Thank you for the detailed observation, and we are glad to hear that you are considering a submission. We warmly welcome externally contributed runs. If you would like to submit results, please run the official RCB evaluation workflow, package the complete run folder, and contact us at xu_wanghan@sjtu.edu.cn.

Regarding the evaluation mode, we agree that this reflects a limitation of using an LLM judge: the judge may occasionally classify the same rubric item differently when evaluating different runs. This is a relatively loose part of the current judging process.

At the same time, we are cautious about changing the scoring prompt or hard-coding the mode at this stage, because doing so may change the scoring distribution for future submissions and affect benchmark consistency and fairness across existing and future results.

A more conservative mitigation may be to run the LLM judge multiple times and average the scores, which can reduce judge-side variance without changing the underlying rubric definition or scoring prompt. We will consider this direction for future scorer calibration.

Thank you again for raising this issue.