Summary
As part of the 2015 SPIE Medical Imaging Conference, SPIE – with the support of American Association of Physicists in Medicine (AAPM) and the National Cancer Institute (NCI) – will conduct a “Grand Challenge” on quantitative image analysis methods for the diagnostic classification of malignant and benign lung nodules. The LUNGx Challenge will provide a unique opportunity for participants to compare their algorithms to those of others from academia, industry, and government in a structured, direct way using the same data sets.
LUNGx Challenge Format
Release of the calibration dataset (with truth): November 21, 2014
- 10 contrast-enhanced CT scans will be available as a calibration dataset. This dataset is representative of the technical properties (scanner type, acquisition parameters, file format) of the test dataset. Participants should not necessarily consider the lung nodules present in the calibration cases to be representative of the difficulty level expected in the test set.
- The calibration set contains five CT scans with malignant nodules and five CT scans with benign nodules. The organizers have selected a single nodule per CT scan for analysis.
- The location of each nodule is specified in an associated Excel file that includes case name, the coordinates of the approximate nodule centroid, and the ‘truth’ label (malignant or benign).
- Participants are encouraged to calibrate their algorithms by downloading the calibration dataset through the TCIA.
Dataset Download
Coming Soon
Disclaimer: Anyone wishing to use the downloaded images for presentation or publication purposes outside of the LUNGx Challenge should acknowledge the SPIE, the NCI, the AAPM, and The University of Chicago:
“Data used in this research were obtained from The Cancer Imaging Archive (TCIA) sponsored by the SPIE, NCI/NIH, AAPM and The University of Chicago.”
Release of the test dataset (without truth): January 9, 2015
Approximately 60 contrast-enhanced CT scans will be available. Again a single nodule will be selected per CT scan. The location of the nodule will be specified in the accompanying Excel file that will follow the same format as for the calibration set with the omission of the ‘truth’ labels.
Dataset Download
This download will be activated on January 9, 2015.
Disclaimer: Anyone wishing to use the downloaded images for presentation or publication purposes outside of the LUNGx Challenge should acknowledge the SPIE, the NCI, the AAPM, and The University of Chicago:
“Data used in this research were obtained from The Cancer Imaging Archive (TCIA) sponsored by the SPIE, NCI/NIH, AAPM and The University of Chicago.”
Deadline for participants to submit test set classification results: February 6, 2015
- Participants should submit their test set classification results by e-mail to Karen Drukker (kdrukker@uchicago.edu).
- In the e-mail subject line it should be stated: 'LUNGx Challenge' .
- An acknowledgment will be sent within 2 business days of receipt of results. In case no confirmation of receipt is received please contact customerservice@spie.org.
Output format for the LUNGx Challenge test set classification results:
In order to facilitate performance assessment of your method for the test cases in the LUNGx Challenge (and to compare your performance to that of the methods of other participants), please follow these guidelines:
- Using your method, calculate a ‘score’ for each case (nodule) assessing the level of suspicion that this case is malignant (or benign).
- Note that the performance of your method will be assessed using Receiver Operator Characteristic (ROC) analysis with the partial area under the ROC curve as a figure of merit. Hence, a continuous scale of your output ‘score’ is preferred (for example, continuous from 0 to 1). A binary ‘score’ (such as 0=benign and 1=malignant with nothing in between) is NOT acceptable. Categorical output is not desirable but may be acceptable if a sufficiently large number of categories in a logical order is used.
- Tabulate your ‘scores’ for all cases in a file containing 2 columns with
- The first column being the case name, the second column being your ‘score’. Example:
- Scan_1_n1 0.567
- Scan_2_n1 0.207
- The preferred file format is Excel, but CSV and OpenOffice (.sxc) also are acceptable.
- The first column being the case name, the second column being your ‘score’. Example:
- Provide a note at the top of your tabulated list whether, for your method, high or low values for the ‘score’ are indicative of malignancy. For example, if your ‘score’ is a probability of malignancy then a higher ‘score’ is expected for actually malignant cases, and hence you would note that ‘high’ values indicate malignancy.
Organizers and Major Contributors:
- Samuel G. Armato, University of Chicago (s-armato@uchicago.edu)
- Lubomir Hadjiiski, University of Michigan Health System (lhadjisk@umich.edu)
- Georgia Tourassi, Oak Ridge National Lab. (tourassig@ornl.gov)
- Karen Drukker, University of Chicago (kdrukker@uchicago.edu)
- Maryellen Giger, University of Chicago (m-giger@uchicago.edu)
- George Redmond, NIH/NCI (gr34m@nih.gov)
- Laurence Clarke, NIH/NCI (lclarke@mail.nih.gov)
- Keyvan Farahani, NIH/NCI (farahank@mail.nih.gov)
- Justin Kirby, Leidos Biomedical, Inc (kirbyju@mail.nih.gov)
- Angela Keyser, AAPM (akeyser@aapm.org)
- Diane Cline, SPIE (diane@spie.org)
- Sandy Hoelterhoff, SPIE (sandyh@spie.org)