Summary

As part of the 2015 SPIE Medical Imaging Conference, SPIE – with the support of American Association of Physicists in Medicine (AAPM) and the National Cancer Institute (NCI) – will conduct a “Grand Challenge” on quantitative image analysis methods for the diagnostic classification of malignant and benign lung nodules. The LUNGx Challenge will provide a unique opportunity for participants to compare their algorithms to those of others from academia, industry, and government in a structured, direct way using the same data sets.

• Release date of calibration set cases with truth:  November 21, 2014
• Release date of test set cases without truth:  January 9, 2015
• Submission date for participants to submit test set classification results:  February 6, 2015
• SPIE Medical Imaging meeting:  February 21 to 26, 2015

 For more information please refer to the LUNGx Challenge Format section listed below.

Challenge participants who submit test set classification results by the February 6 deadline are also invited to present poster presentations during the conference and may demonstrate their algorithms during the live demonstration CAD workshop on Tuesday evening [February 24]. Also, a special panel discussion on Tuesday afternoon will focus on the topic, and the winner of the Challenge will join the special panel discussion and receive a waiver of the conference registration fee for SPIE MI 2015.

Participants are encouraged to submit their work to the SPIE CAD proceedings volume as well as for peer review to the new SPIE Journal of Medical Imaging.

LUNGx Challenge Format

Release of the calibration dataset (with truth):  November 21, 2014

• 10 contrast-enhanced CT scans will be available as a calibration dataset. This dataset is representative of the technical properties (scanner type, acquisition parameters, file format) of the test dataset. Participants should not necessarily consider the lung nodules present in the calibration cases to be representative of the difficulty level expected in the test set.
• The calibration set contains five CT scans with malignant nodules and five CT scans with benign nodules. The organizers have selected a single nodule per CT scan for analysis.
• The location of each nodule is specified in an associated Excel file that includes case name, the coordinates of the approximate nodule centroid, and the ‘truth’ label (malignant or benign).
• Participants are encouraged to calibrate their algorithms by downloading the calibration dataset through the TCIA.

Release of the test dataset (without truth):  January 9, 2015

Approximately 60 contrast-enhanced CT scans will be available. Again a single nodule will be selected per CT scan. The location of the nodule will be specified in the accompanying Excel file that will follow the same format as for the calibration set with the omission of the ‘truth’ labels.

Deadline for participants to submit test set classification results:  February 6, 2015

• Participants should submit their test set classification results by e-mail to Karen Drukker (kdrukker@uchicago.edu).
• In the e-mail subject line it should be stated:  'LUNGx Challenge' .
• An acknowledgment will be sent within 2 business days of receipt of results. In case no confirmation of receipt is received please contact customerservice@spie.org.

Output format for the LUNGx Challenge test set classification results:

In order to facilitate performance assessment of your method for the test cases in the LUNGx Challenge (and to compare your performance to that of the methods of other participants), please follow these guidelines:

1. Using your method, calculate a ‘score’ for each case (nodule) assessing the level of suspicion that this case is malignant (or benign).
   1. Note that the performance of your method will be assessed using Receiver Operator Characteristic (ROC) analysis with the partial area under the ROC curve as a figure of merit. Hence, a continuous scale of your output ‘score’ is preferred (for example, continuous from 0 to 1). A binary ‘score’ (such as 0=benign and 1=malignant with nothing in between) is NOT acceptable. Categorical output is not desirable but may be acceptable if a sufficiently large number of categories in a logical order is used.
2. Tabulate your ‘scores’ for all cases in a file containing 2 columns with
   1. The first column being the case name, the second column being your ‘score’. Example:
      1. Scan_1_n1       0.567
      2. Scan_2_n1       0.207
   2. The preferred file format is Excel, but CSV and OpenOffice (.sxc) also are acceptable.
3. Provide a note at the top of your tabulated list whether, for your method, high or low values for the ‘score’ are indicative of malignancy.  For example, if your ‘score’ is a probability of malignancy then a higher ‘score’ is expected for actually malignant cases, and hence you would note that ‘high’ values indicate malignancy.

Organizers and Major Contributors:

• Samuel G. Armato, University of Chicago (s-armato@uchicago.edu)
• Lubomir Hadjiiski, University of Michigan Health System (lhadjisk@umich.edu)
• Georgia Tourassi, Oak Ridge National Lab. (tourassig@ornl.gov)
• Karen Drukker, University of Chicago (kdrukker@uchicago.edu)
• Maryellen Giger, University of Chicago (m-giger@uchicago.edu)
• George Redmond, NIH/NCI (gr34m@nih.gov)
• Laurence Clarke, NIH/NCI (lclarke@mail.nih.gov)
• Keyvan Farahani, NIH/NCI (farahank@mail.nih.gov)
• Justin Kirby, Leidos Biomedical, Inc (kirbyju@mail.nih.gov)
• Angela Keyser, AAPM (akeyser@aapm.org)
• Diane Cline, SPIE (diane@spie.org)
• Sandy Hoelterhoff, SPIE (sandyh@spie.org)