==== Title: High Throughput Truthing (HTT): Pathologist Agreement from a Pilot Study ====

==== Abstract ====

Background

* Artificial intelligence algorithms in digital pathology have enormous potential to increase diagnostic speed and accuracy. However, the performance of these algorithms must be validated against a reference standard before deployment in clinical practice. In this work, pathologists are considered as the reference standard. We studied interobserver variability in pathologists who evaluate stromal tumor-infiltrating lymphocytes (sTILs) in hematoxylin and eosin stained breast cancer biopsy specimens. Our ultimate goal is to create a validation dataset that is fit for a regulatory purpose.

Methods

* Following an IRB exempt determination protocol, we obtained informed consent of volunteer pathologist annotators prior to completing data collection tasks via two modalities: an optical light microscope and two digital platforms (slides were scanned at 40X). Pathologists were trained on the clinical task of sTIL density estimation before annotating pre-specified regions of interest (ROIs) across multiple platforms. The ROI selection protocol sampled ROIs in the tumor, tumor boundary, and elsewhere. Inter-pathologist agreement was characterized with the root mean-squared difference, which is analogous to the root mean-squared error but doesn’t require ground truth.

Results

* The pilot study accumulated 6,257 sTIL density estimates from 34 pathologists evaluating 64 cases, with 10 ROIs per case. The variability of sTIL density estimates in an ROI increases with the mean; the square roots of the reader-averaged mean-squared differences were 8.3, 17.7, and 40.4 as the sTIL density reference score increased from 0-10%, 10-40%, and 40-100%, respectively. We also found that the root mean-squared differences for some pathologists were considerably larger than others (as much as 120% larger than the next largest root mean-squared difference).

Conclusions

* Slides, images, and annotations were successfully provided by volunteer collaborators and participants, which created an innovative and thorough method for data collection and truthing. This pilot study will inform the development of statistical methods, simulation models, and sizing analyses for pivotal studies. The development and results of this validation dataset and analysis tools will be made publicly available to serve as an instructive tool for algorithm developers and researchers. Furthermore, the methods used to analyze pathologist agreement between density estimates are applicable to other quantitative biomarkers.