Study FDA-MSKCC 14-head Microscope

by Brandon D. Gallas

On Nov. 3, 2017 Brandon Gallas and Qi Gong visited MSKCC to help Yukako Yagi conduct a study using eeDAP on a 14-head microscope. There were 11 pathologists participating. There were 40 regions of interest (ROIs) where they were asked to detect and mark the locations of mitotic figures. For each ROI, there were candidate mitotic figures identified during a preliminary study. There were 126 candidate mitotic figures in total. The participants were asked to classify these candidates as mitotic figures or not. The study was completed in 2 hours!

Resulting data:

  • 12 participants (11 pathologists + 1 non-pathologist; 1 pathologist did not complete)
  • Count data: count the number of mitotic figures in ROI
    • 4 slides, 10 ROIs per slide
  • Mark data: participants marked mitotic figures (circle on paper)
    • 4 slides, 10 ROIs per slide
  • Classification data: participants scored candidate mitotic figures from 0-100
    • 126 candidate mitotic figures
  • Data was collected on paper. We used eeDAP to transcribe this data to electronic form. For the mark data, this was accomplished with ImageScope integrated within the eeDAP work flow.
    • eeDAP writes an xml file with annotations of the FOV and the ROI.
    • eeDAP calls ImageScope, which refers to xml file created as above.
    • user adds annotation to WSI within ImageScope, and then saves and closes ImageScope.
    • eeDAP moves updated xml to eeDAP output directory.

The counts and classifications were done without discussion under instructions to make decisions independently, and the classifications were confidence scores as opposed to binary yes/no classifications: a 0-100 scale, threshold=50. We believe that there is more information in multi-level scores that can lead to smaller studies or better statistical power. The candidate MFs were identified during a preliminary image curation study where 5 pathologists counted and marked the locations of MFs arising from the same 40 ROIs. We call them “candidate” MFs because only 21 of 92 were unanimously identified. The distribution of agreement results across the candidates reflects a spectrum of candidates from obvious and likely MFs to subtle and improbable MFs. In fact, in order to include candidate MFs with even lower pathologist agreement, we asked one of the image curation pathologists to re-evaluate 12 of the original 40 ROIs. These ROIs had only one candidate or none identified during the original image curation. For this re-evaluation we asked the image curation pathologist to lower his threshold for identifying MFs. He identified 36 more candidates. This spectrum allows an efficient and effective evaluation of an AI algorithm.

Image Curation Data

  • 4 pathologists (1 pathologist joined later)
  • Count data: count the number of mitotic figures in ROI
    • 4 slides, 10 ROIs, digital and micoRT
  • Mark data: participants marked mitotic figures (circles on pdf)
    • 4 slides, 10 ROIs, circles on pdf based on digital mode
    • Yielded 92 candidates (34 more candidates identified later at lower threshold)

Study Materials

  • Scroll down to see a picture of all the pathologists working on the 14-head microscope.
  • Here is a link to our original plans.
  • Here is an abstract submitted (but not accepted) to the USCAP 2018 annual meeting. It outlines the basics: the preliminary study and the 14-head study. USCAP2018abstractGallas3-submitted.docx (7 MB, uploaded by Brandon D. Gallas 4 years 1 week ago)
  • Here is a summary of the study written for the participants. FDA-MSKCCmultiHead-2.docx (25 KB, uploaded by Brandon D. Gallas 4 years 1 week ago), with some revisions FDA-MSKCCmultiHead-3.docx.
  • Here is a picture of the instructions on the white board during the study Link to Pic
  • Here is the case report form. We used paper this time, hoping to have a cloud solution next time. caseReportForm-v2.docx (6 MB, uploaded by Brandon D. Gallas 4 years 1 week ago)
  • Here are quick-access instructions for running the preliminary study. instructions_mcsMSKCC.docx (2 MB, uploaded by Brandon D. Gallas 4 years 1 week ago)
  • Here are instructions for running eeDAP with ImageScope to collect image annotations. saveImageScopeAnnotationsTaskWorkflow.docx (17 KB, uploaded by Brandon D. Gallas 4 years 1 week ago)
  • Here is the definition of a mitotic figure and the region of interest used for the studies. mitosisDefinition.docx (67 KB, uploaded by Brandon D. Gallas 4 years 1 week ago)
  • Here is the observer consent form for the study. observerConsentForm_mcsMSKCC-2.docx (28 KB, uploaded by Brandon D. Gallas 4 years 1 week ago)
  • Here is the observer registration form. readerRegistration-2.docx (19 KB, uploaded by Brandon D. Gallas 4 years 1 week ago)

mskcc14headStudy.png

Created on , Last modified on