26 July 2021 Update
We are pleased to announce some recent accomplishments of the High-throughput truthing project (HTT).
The HTT project’s objective is to create a validation dataset fit for a regulatory purpose. Pursuing this objective is expected to inform regulatory frameworks and be instructive to others to develop their own validation datasets. For more detailed information about the project and/or to register as a pathologist-data-collector, please visit https://ncihub.org/groups/eedapstudies/.
Please contact Katherine Elfer (Katherine.Elfer@fda.hhs.gov), Project Coordinator, or Brandon Gallas (Brandon.Gallas@fda.hhs.gov), Project Lead, with any questions or to join our efforts.
Upcoming activities
9/3/2021: Discussion about patient sampling for HTT pivotal study and CME Training
- Lead: Victor Garcia, MD, is a new FDA fellow, board certified in internal medicine and recently completed a Clinical Informatics Fellowship.
Summer 2021-Fall 2021: Recruiting Pathologists for on-site Data Collection
- We are looking for pathologists to provide annotations using the eeDAP system
- Data Collection Sites; Yale New Haven
- LINK to full recruitment and information
Recent accomplishments
8/19/2021: FDA Summer Intern Festival – Google Summer of Code Video
- 2021 Google Summer of Code Intern Peddi Sai Varshith recorded a video of his work with the High Truthing Project and integrating hardware with caMicroscope
- Video and Project Summary: HERE
8/12/2021: FDA Annual Summer Student Scientific Poster Day
- 9:30 – 11:30am EDT
- Two Posters from our summer interns:
- Title 1: Developing a controlled workflow for clinical studies to collect pathologist annotations
- Title 2: Communication Methods and Workflows to Develop and Implement High Throughput Truthing of Pathologist Annotations as a Reference Standard for Validating Artificial Intelligence in Digital Pathology.
- Link to posters and full abstracts
- Link to Register for full FDA Webcast & Poster Event
8/8/2021: Public Release of the HTT !Github Repository
- Contains primary data objects (with documentation) and supporting functions
- Contains Pathology Informatics R markdown file with code to process data and create all figures
- https://github.com/DIDSR/HTT
8/8/2021: Presentation at the American Statistical Society Joint Statistical Meetings
- “Pathologist Agreement from Quantitative Measurements: a Pilot Study”
- In coordination with this presentation and the corresponding proceedings paper, we intend to make public a GitHub repository (name “HTT”) with an R data package containing the pathologist annotations.
- LINK to Full Recordings and Abstract
- LINK to Conference Program
7/2021: Statistical Methods Presentation
- Si Wen, Statistician FDA/CDRH/OSEL/Division of Imaging, Diagnostics, and Software Reliability
- Title: Quantitative Agreement Analysis for HTT Pilot Study Data
- Watch HERE
- GitHub repository with methods: https://github.com/DIDSR/ANOVA.MRMC.LOA
- Related manuscript: Preprint on arXiv – 2107.08891. It has been submitted to Statistics in Biopharmaceutical Research.
7/2021: Shipped eeDAP system to Yale for microscope-mode data collection
- eeDAP Github Repo
- Microscope professionally cleaned and ready for data collection
- Looking for pathologists to provide annotations
- Upgraded eeDAP hardware for speed
- Resolved third-party software version conflicts (Bioformats, Java, and Matlab – https://github.com/DIDSR/eeDAP/issues/89 )
7/2021: We have executed two research collaboration agreements (RCA) for slide sourcing.
- We are looking for at least one other US site for slide sourcing
- We have an established RCA template
- LINK to RCA Abstracts and Template
6/2021: Hired five summer research assistants to support related research
- Victor Garcia, MD MS: Stony Brook Medicine
- Sarah Dudgeon, MPH (back to join us for data collection!): CORE Center for Computational Health New Haven Hospital
- John Paul (JP) Philips: University of Chicago, Dept. of Physics
- Phoebe Qian: Mc.Clean High School, VA
- James Setty: Wall High School, D.C.