Presentation 2022: Pathology Innovation Collaborative Community Webinar - ROC and MRMC Tutorials - Gallas
- Version 2
- by (unknown)
- Version 3
- by (unknown)
Deletions or items before changed
Additions or items after changed
| 1 | = [https://ncihub.org/groups/eedapstudies/wiki/PresentationsandPublications Back to Presentations and Publications Main Page] = | |||
|---|---|---|---|---|
| 2 | ||||
| 3 | = Pathology Innovation Collaborative Community Webinar = | |||
| 4 | * Webinar | |||
| 5 | * 12:00 PM, Friday, 5 August 2022 | |||
| 6 | + | |||
| 7 | + | This presentation was also give as FDA internal training on 8 April 2022.
|
||
| 8 | ||||
| 9 | ==== Title 1: Tutorial on Reader Study Designs and MRMC Analysis ==== | |||
| 10 | ==== Title 2: ROC curves: Receiver Operating Characteristic Curves ==== | |||
| 11 | ||||
| 12 | ==== Slides ==== | |||
| 13 | * [[File(20220805-PIcc-MRMCstudyDesigns-Gallas.pdf)]] | |||
| 14 | * [[File(20220805-PIcc-ShortROCtutorial.pdf)]] | |||
| 15 | ||||
| 16 | ==== Videos ==== | |||
| 17 | * [https://vimeo.com/751682643 Video MRMC tutorial: 1 hour 20 seconds] | |||
| 18 | * [https://vimeo.com/751670299 Video ROC tutorial: 6 minutes 17 seconds] | |||
| 19 | ||||
| 20 | ==== Abstract ==== | |||
| 21 | In this talk I will present the major elements to design, execute, and analyze reader studies. In a reader study, clinicians perform an objective task given medical images. The purpose of a reader study is to assess the clinicians’ performance doing the task given the images or to compare performance given images from a new technology to the performance given images from the reference technology. The clinician is part of the technology assessment. The clinicians are the readers and the medical images are the cases. The objective tasks are to make diagnostic evaluations or other measurements given the images (aligned with the intended use of the images). Some refer to the evaluations as subjective because they involve humans who are biased and not fully reproducible. I prefer to refer to the evaluations as objective to distinguish them from evaluations that are, in fact, opinions that have no right answer. An objective evaluation can be compared to truth if truth is available. Of course, there is bias and variability from human evaluations. As such, analysis methods for reader performance (variance estimates, confidence intervals, hypothesis tests) need to account for such bias and variability from the readers while they also account for the bias and variability from the cases. Such an analysis is referred to as a multi-reader, multi-case MRMC analysis. An MRMC analysis can summarize average reader performance that is expected to generalize to the population of readers and the population of cases. | |||
| 22 | ||||
| 23 | ==== Author ==== | |||
| 24 | Brandon D. Gallas, !PhD | |||
| 25 | * Division of Imaging, Diagnostics, and Software Reliability, OSEL/CDRH/FDA, White Oak, MD | |||