30 September 2021 Update
We are pleased to announce some recent accomplishments of the High-throughput truthing project (HTT).
The HTT project’s objective is to create a validation dataset fit for a regulatory purpose. Pursuing this objective is expected to inform regulatory frameworks and be instructive to others to develop their own validation datasets. For more detailed information about the project and/or to register as a pathologist-data-collector, please visit https://ncihub.org/groups/eedapstudies/.
To contact us with any questions or join our efforts: Contact Us HERE
Current and upcoming activities
10 Sept-1 Oct: Joint TILs Annotation and Discussion Sessions with Pathologists
- Dr. Victor Garcia and Dr. Brandon Gallas are leading discussions to identify critical instructions for our annotation trainings that will lead to less pathologist variability (generally) and improve our future validation dataset (specifically).
- We’d be happy to circulate summaries of the discussions and the feedback we will give to pathologists.
- We plan to present the results at Pathology Informatics and in a publication.
Summer 2021-Fall 2021: Recruiting Pathologists for on-site Data Collection
- We are looking for pathologists to provide annotations using the eeDAP system
- Data Collection Sites; Yale New Haven
- LINK to full recruitment and information
Recent accomplishments
27 September: Dr. Kate Elfer was a panelist concluding the 2021 Computational Pathology Workshop
- 2021 Computational Pathology (COMPAY21) workshop part of the 24th International Conference on Medical Image Computing and Computer Assisted Intervention
- https://www.examode.eu/compay2021/
- Panelists: Katherine Elfer (FDA); Chen Sagiv (DeePathology.ai); Roberto Salgado (TIL Working Group); Inti Zlobec (University of Bern); Faisal Mahmood (Harvard Medical School); Monika Lamba Saini (CellCarta)
9/9/2021: Dr. Brandon Gallas presented to the PathLake Conference at Pathology Horizons
- Title: Regulatory Mechanisms and Tools for Software as a Medical Device
- Includes short discussion about algorithm change control plans
- Slides
9/3/2021: Discussion about patient sampling for HTT pivotal study and CME Training
- Lead: Victor Garcia, MD, is a new FDA fellow, board certified in internal medicine and recently completed a Clinical Informatics Fellowship.
8/19/2021: FDA Summer Intern Festival – Google Summer of Code Video
- 2021 Google Summer of Code Intern Peddi Sai Varshith recorded a video of his work with the High Truthing Project and integrating hardware with caMicroscope
- Video and Project Summary: HERE
8/12/2021: FDA Annual Summer Student Scientific Poster Day
- 9:30 – 11:30am EDT
- Two Posters from our summer interns:
- Title 1: Developing a controlled workflow for clinical studies to collect pathologist annotations
- Title 2: Communication Methods and Workflows to Develop and Implement High Throughput Truthing of Pathologist Annotations as a Reference Standard for Validating Artificial Intelligence in Digital Pathology.
- Link to posters and full abstracts
- Link to Register for full FDA Webcast & Poster Event
8/8/2021: Public Release of the HTT !Github Repository
- Contains primary data objects (with documentation) and supporting functions
- Contains Pathology Informatics R markdown file with code to process data and create all figures
- Repository: https://github.com/DIDSR/HTT
8/8/2021: Presentation at the American Statistical Society Joint Statistical Meetings
- “Pathologist Agreement from Quantitative Measurements: a Pilot Study”
- In coordination with this presentation and the corresponding proceedings paper, we intend to make public a GitHub repository (name “HTT”) with an R data package containing the pathologist annotations.
- LINK to Full Recordings and Abstract
- LINK to Conference Program
7/2021: Statistical Methods Presentation
- Si Wen, Statistician FDA/CDRH/OSEL/Division of Imaging, Diagnostics, and Software Reliability
- Title: Quantitative Agreement Analysis for HTT Pilot Study Data
- Watch HERE
- GitHub repository with methods: https://github.com/DIDSR/ANOVA.MRMC.LOA
- Related manuscript: Preprint on arXiv – 2107.08891. It has been submitted to Statistics in Biopharmaceutical Research.
7/2021: Shipped eeDAP system to Yale for microscope-mode data collection
- eeDAP Github Repo
- Microscope professionally cleaned and ready for data collection
- Looking for pathologists to provide annotations
- Upgraded eeDAP hardware for speed
- Resolved third-party software version conflicts (Bioformats, Java, and Matlab – https://github.com/DIDSR/eeDAP/issues/89 )
7/2021: We have executed two research collaboration agreements (RCA) for slide sourcing.
- We are looking for at least one other US site for slide sourcing
- We have an established RCA template
- LINK to RCA Abstracts and Template
6/2021: Hired five summer research assistants to support related research
- Victor Garcia, MD MS: Stony Brook Medicine
- Sarah Dudgeon, MPH (back to join us for data collection!): CORE Center for Computational Health New Haven Hospital
- John Paul (JP) Philips: University of Chicago, Dept. of Physics
- Phoebe Qian: Mc.Clean High School, VA
- James Setty: Wall High School, D.C.