Agreement is measurable only where two raters saw the same claim. In this pilot that is the 10 claims the two complete raters share, and nowhere else. The reference plotted against it is not the naive one-in-four line but the agreement each rater’s own selection habits already produce by chance, derived from their observed marginals.
The objection to the ordering is that two people simply agreed with each other. If that were what produced it, agreement between these two would have to sit above what each rater's own selection habits already generate by chance. Here it lands at or below that line on MOST and LEAST.
Expected agreement under independence, conditional on the two observed marginal distributions — an appropriate chance correction, not uniquely “the correct” one at 10 shared claims. Read the hollow mark as one defensible reference among several, not as the answer. src/ers_evidence/agreement.py
2 raters over 10 shared claims is a very small base. A κ below zero here is not evidence of systematic disagreement; it is what small samples do. This weakens one alternative explanation of the ordering — that it came from these two agreeing item by item — and nothing more. It says nothing about the arms.
That benchmark is one defensible chance correction at this size, not uniquely the correct one, and
src/ers_evidence/agreement.py says so where the quantity is computed. The hollow mark is a
reference, not a threshold.
At 10 shared claims from 2 raters the quantity is unstable by construction — a single changed selection moves it. A kappa below zero at this size is what small samples do, not evidence that these raters systematically disagree.
The comparison bears on one objection and no others. If the aggregate ordering were an artifact of two people agreeing item by item, their agreement would have to clear what their own habits already generate. On this pair it does not. That weakens the objection; it does not settle it, and it says nothing about the arms.