40. Testing Citation Systems and Measuring Improvement

40.1 Define what success means

Test whether readers can identify the right source form, retrieve the source, locate the passage, distinguish its role, and detect a version or status issue. Measure correctness before speed. A fast route to the wrong source is a failure even if users prefer its appearance.

For a reference book, useful measures include time to a correct answer, percentage of tasks completed accurately, unresolved-source rate, and severity of errors. Record the participant group and task difficulty. Do not generalize from a handful of familiar examples to every jurisdiction or source type.

40.2 Separate tool capabilities

Extraction finds citation-like text. Identity matching associates a citation with a source. Locator checking tests the passage address. Support review evaluates the proposition. Treatment research examines later developments. A tool that performs one stage should not display a blanket “legally correct” result. Preserve uncertainty at the stage where it arises.

Our source research provides a concrete example: a batch analyzer matched six reporter citations but attached several preceding names to subsequent citations. Individual checks confirmed the identities. A system should retain that warning, investigate it, and record the resolution; neither automatic acceptance nor automatic rejection would explain the actual result.

40.3 Design meaningful test cases

Include correct citations, near matches, ambiguous short forms, wrong courts, missing years, amended opinions, separate writings, broken links, and unavailable sources. Keep fictional test fixtures labeled. A synthetic test can establish program behavior for that fixture; it cannot certify the upstream legal database or a real authority's status.

40.4 Compare references fairly

Use the same tasks, receiving instructions, time limits, and scoring method for each resource. Include unfamiliar tasks and several user experience levels. Report the sample and uncertainty. “Better” can mean broader coverage, faster retrieval, fewer mistakes, clearer explanation, or lower cost; specify the dimension rather than claiming an undefined multiple of superiority.

40.5 Maintain a release ledger

For each edition, record manuscript version, source baseline, completed checks, unresolved coverage, and corrections. Distinguish structural testing from independent legal review and reader testing. This expanded edition supplies original chapters, source forms, receiving-rule profiles, jurisdiction gateways, and reusable worksheets. The ledger identifies which parts are inspected and which require further review.

Publication standard. A broad reference can be useful while openly documenting its limits. Its next edition should be driven by observed reader failures, source changes, and substantive review, not by page count alone. The promise is a maintainable method and inspectable evidence, with improvements that can be demonstrated.