This user's guide provides the basic guidance for Image Provider sites to submit new collections to The Cancer Imaging Archive (TCIA). It summarizes the overall workflow for the entire process and covers in detail the installation and use of the Clinical Trials Processor (CTP) Client software. This is the primary tool used to de-identify and upload images into The Cancer Imaging Archive.
Use this guide in conjunction with The Cancer Imaging Archive User's Guide. A collections facilitator at the University of Arkansas for Medical Sciences (UAMS) will be assigned to assist the staff at the Image Provider site in installing the necessary software and dealing with technical issues. Questions about the use of this guide should be sent to The Cancer Imaging Archive User Help Desk at firstname.lastname@example.org.
Overview of the Submission Process
As a provider of image data to The Cancer Imaging Archive, you can expect a collections facilitator from The Cancer Imaging Archive (TCIA) team to provide all the required tooling and answer your questions throughout the process.
You will be asked to:
- discuss your collection with the image submission team and establish a timeline for uploading your image collection;
- install the custom-configured CTP software suite that will be provided and use it to submit your image collection to The Cancer Imaging Archive.
You will receive:
- A custom-configured CTP software suite that will provide local de-identification at your site, following the DICOM WG18 Supplement 142 standard. CTP will submit the data to The Cancer Imaging Archive over the https protocol so that no firewall ports will need to be opened at your site.
- Quality control and curation staff that will work with you to ensure that the data are de-identified according to the DICOM Supplement142 standard and that all data are received.
- The option of working interactively with your assigned collections facilitator via a convenient web meeting tool.
The Cancer Imaging Archive team will work with you every step of the way to make sure the upload process goes smoothly and your valuable contribution is accurately and completely incorporated into The Cancer Imaging Archive.
Take the First Step: Request Permission to Upload Your Data
The Cancer Imaging Archive is intended to be a resource to the wider imaging research community. Proposals for additional data submissions are welcomed and encouraged. The CIP informatics team will be glad to work with you to evaluate the suitability of your data to this archive.
If you feel that you have data at your institution that meets one or more of these objectives, please fill out our application form so that we can evaluate the suitability of your data to the archive. Questions about filling out the application may be directed to the help desk.
Our goal in providing this service is to ensure that every image collection in the archive is one that would be of value to one of our target audiences. These include researchers with the following objectives:
- Meeting the data sharing requirements set forth by an NCI grant award
- Sharing data for analyzing imaging features to be used as biomarkers
- Sharing data for comparing image features to other data types such as genetics, pathology, or clinical information to create correlative signatures as biomarkers
- Sharing data for the creation of automated or semi-automated algorithms for detection of cancer
- Sharing data as a reference collection for testing and validating quantitative analysis techniques or algorithms in image processing
To determine if your image collection meets these requirements or to discussion potential submissions, contact the TCIA Help Desk.
Analyzing Your Data Prior to Submission
In order to properly de-identify your collection during the submission process it is essential that The Cancer Imaging Archive team completely understands the unique characteristics of your data. Since all images uploaded to The Cancer Imaging Archive use the DICOM standard defined image formats, it is essential to understand any vendor-specific private data elements that may contain Protected Health Information (PHI). The Private Tag Research Procedure and TagSniffer software package have been created to support a complete analysis of your unique image collection.
The Clinical Trials Processor (CTP) is the primary means for uploading data into The Cancer Imaging Archive. CTP is a client-server software package with the server installed on a special intake system at UAMS. You will be given a pre-configured CTP client, containing de-identification scripts customized for your site and collection. You will receive instructions on the proper installation and usage of CTP, including those below as well as additional instructions provided by the collections facilitator.
Checking for Java Programs
The CTP software is written in the Java programming language so that it can be supported on all common computing platforms and operating systems.
Check for Java
Before installing CTP Client, the Java 1.6.0_24 (or better) Java Runtime Environment (JRE) must be present on the system. Do the following to check for the current version on your PC.
Only the JRE is required, not the JDK.
- Go to the Java website and verify your Java version.
If the response indicates JRE is not present or the version is older than 1.6.0_24, install Java or a new version of Java.
Check for Java Image I/O tools (usually not required)
Certain CTP pipeline stages (FileStorageService, BasicFileStorageService) require that the Java Advanced Imaging ImageIO Tools version 1.1 be present on the system. These are only necessary if you are retaining a copy of the images on your local computer. The assigned collections facilitator for your site will inform you if Image IO Tools are required. Do the following to check for the current version installed on your PC.
- Click Start > Control Panel > Programs and Features.
- Scroll down the list to find the installed Java programs.
If your installed version of JAI Tools is 1.0, install the latest version of JAI Tools.
JAI Tools download and install instructions
Certain CTP pipeline stages (FileStorageService, BasicFileStorageService) require that the Java Advanced Imaging ImageIO Tools be present on the system. It is critically important that version 1.1 of the ImageIO Tools be installed rather than version 1.0. Parenthetically, note that the Java Advanced Imaging component is not the same as the Java Advanced Imaging ImageIO Tools. Only the latter component is required. The ImageIO Tools need not be present in order to run the installer, and they may be installed later if required, without having to re-install CTP. From the download page linked to above, select the appropriate JRE version for your operating system and download to your hard drive. Install and reboot if requested.
Installing CTP Client
The collections specialist will provide each Image Provider site with a compressed file containing the CTP program files and configuration files customized for each site and collection. This file should be copied to a directory on the computer that will run the CTP program. Follow these steps to install the CTP Client software.
- Double-click the file and select Open.
- Use the Windows Extraction Wizard or a similar program to uncompress the files.
- The recommended location for the CTP Client files is in a folder off the root directory. During the unzipping process you can create a folder such as C:\CTP Client by entering the new folder name in the text box. Click Next.
- Click Finish.
Starting CTP Client
Follow these steps to start CTP Client:
- Run (double click) the CTP-launcher.jarfile found in C:\CTP Client\CTP folder.
- In the CTP Launcher window, click the Start button and then click the CTP Home Page button. This opens the CTP Home Page in your web browser.
- Select Login from the home page and then log in using 'admin' and the password 'password'.
FileSender is a utility program for sending files to CTP. It is installed as part of the CTP installation process. It is used when image files are in multiple directories. Follow these steps to use FileSender:
- Start CTP.
- Launch the FileSender program by double-clicking the FileSender.jarfile in the C:\CTP Client\File Sender folder.
The left pane in FileSender is used to navigate to a file or a folder of interest. The pull-down menu in the header of the left pane lets you choose the root (drive) file system. The button in the footer bar lets you specify the extensions to accept. Enter an asterisk if you wish to accept all files, regardless of their extensions.
Check (by clicking on it) the "Include subdirectories" option, if you want FileSender to drill down into all the subdirectories and send all files.
- The right pane is used to specify the destination URL. Unless advised otherwise always use "dicom://ctp:me@localhost:11112".
- Once a file or folder is selected in the left pane, enter or select a URL in the right pane by clicking its radio button. Start the transmission by clicking the Send button in the footer bar of the right pane.
- Return to CTP and continue monitoring your transmission.
New Collection Submission
Image Transmission Test
The UAMS collections facilitator will work with you to facilitate a limited test-set submission of your collection once the CTP client has been installed. This test will consist of one series of images using either a DICOM 'send' via your PACS or using FileSender to transmit the data from a separate storage system on which you have pre-loaded your data to be submitted. Once the limited test-set has been uploaded you will be asked to send an email to UAMS with the image count. The collections facilitator will verify successful submission of data and provide assistance if there were any errors. Optionally this test will be performed interactively (via web conferencing) by a UAMS collections facilitator.
Uploading Your Collection
Monitoring your transmission
Follow the image transmission process after selecting the folder/image in FileSender (step #5). Image provider sites should monitor progress of the transmissions by checking the CTP Status Page during the transmission.
The files processed count should increase during the transmission. Refresh the browser to check on progress.
- In the Archive Import Service, the Archive Files Supplied should be increasing as you click Refresh.
- At the bottom, the HTTP Export, Queue Size should not be steadily increasing.
- At the completion of the transmission the files processed count should equal the number of image files needed to be sent.
Checking that your transmission completed successfully
After the transmission is complete several pages should be checked to ensure the transmission was completed successfully. For problems, please consult Troubleshooting and Error Reporting.
- Check the Status page. Verify that the collection has been transmitted. The Archive Import Service should show Archival Traversal = complete. The HTTP Export, Queue size should = 0.
- Check All Quarantines. Check individual quarantines if the results do not equal 0.
- Check the database verifier. Note: Scroll down to the bottom of the database verifier. If it does not indicate that "Unverified Queue Size = 0" or if you have any quarantines, please call your collections facilitator. If the unverified count = 0 and you have no quarantines, consider taking a screenshot for your records and for emailing to your collections facilitator.
- Viewing your images. You will be unable to see your images until your collection facilitator's QC team has inspected the images and made them visible. At that point, your collection facilitator will ask if you would like to view them before they are moved to the public server. While you will be able to view them before they are moved to public, you will be unable to download any images until they are public.
Email transmission results
This is critical! The submission process cannot be completed without this step. At the end of a successful transmission, please email your collections facilitator the image count from the status page; a screenshot of the database verifier results is most welcome! Also in the email you are encouraged to include comments, observations, suggestions, or criticisms. These will help us make this process and documentation better.
Alternative Image Transmission Methods
FileSender is the most versatile method of file transmission and will be used in most cases. CTP also provides three additional methods of image transfer. Your submissions facilitator will determine the most appropriate method based on information you provide during initial telephone discussions. Please refer to the following information for other transmission methods.
If the image files are in a single location the 'treeRoot' method of image transmission can be used with no additional steps required. The disadvantage of this method is not being able to select only certain images for transmission.
Sending from a PACS
CTP Client is configured to receive images directly from a PACS. Your PACS administrator will need to configure the PACS to send to the ipaddress of the computer that is running CTP. CTP is configured to listen on port 11112 and will accept all AET connections. Before pushing images from the PACS, start the CTP Client and monitor the submission.
Multiple pipeline submissions from same site
Preserving important files after submission
Image Providers are strongly encouraged to preserve the ID Mapping table for future reference if necessary.
- Select ID Map from the CTP home page.
- Click on the IDMap pipeline.
- Select Original Patient ID from the Select Key Type drop-down and CSV from the Select Return Format drop-down.
- Save the CSV file to a safe place on your hard drive or copy to portable media.
Troubleshooting and Error Reporting
Overview and Analysis of an Issue
When something goes awry, the reason(s) may not be readily apparent. The sections that follow describe procedures for resolving more common issues. If the procedures do not apply to your issue or apply but do not work, please consult your collection facilitator.
The cause of unverified images may be hard to discern. At times, it may be that transmissions and client-server handshakes are still in progress and the unverified counts are merely temporary. If it has only been a few minutes since the last images were transmitted, you should continue to check the unverified counts until they reach zero or stabilize. If the count stabilizes greater than zero, you should check the export queue in the CTP-Client User Interface (UI) to make sure all images have been sent (count = 0). If greater than zero and stable, you should shutdown the CTP-Client and restart it to see if this gets transmissions moving again. If the export queue is zero, but the unverified number is non-zero, then the images either got lost in transmission, or are stuck in a queue or quarantine on the server side. You will need to dialog with your submissions specialist. At other times, unverified images may be related to the submission of duplicate images, for which there may be clues in quarantines. You should provide the collections facilitator with whatever information is known about the unverified images so UAMS can examine server-side logs and quarantines in order to advise submitter how to proceed. One solution may require the submitter to re-transmit images.
- Log on to the CTP-Client and click Quarantines.
- Click on quarantine of interest (one with the number of images > 0).
- You'll see a list of quarantined images, one per line, across 4 columns, where columns are:
- Link to a readable dump of image's DICOM header
- Date and time (image was quarantined)
- Queue button
- Delete button
- Click on any of the links in the left column to see the DICOM header dump. You may print-preview and print to the extent your browser allows.To save the dump, click the Download the file button at the bottom.
- If you want to view the image, you cannot do this from the CTP-Client UI; however, leave the list of images open in one window.In another window, get to the following directory or folder: $PATH/CTP-Client/CTP/quarantinesWhere $PATH specifies the front end of the full path to the CTP-Client. Choose the appropriate subdirectory that matches the type of quarantine in question.The files in this subdirectory should match the list of files in the CTP-Client UI. Launch your favorite image viewer to view the image in question.
- If, after inspecting the image, you want to delete it, click the Delete button in right column.
- If you really want to include this image in the collection, consult your collections specialist. The dialog will focus on what caused the image to be quarantined. UAMS will then provide a modification to the script that caused the quarantine. Submitter would then restart the CTP-Client with new script in place. Submitter would proceed to the previously-chosen quarantine area, and, on the line of the image in question, click "queue". The image would then be re-queued and allowed to pass into submission.
There are two situations in which images might need to be retransmitted. There may have been serious problems with your transmission that require starting over from scratch, or alternatively you may need to retransmit a subset of images to fix a smaller problem.
Starting your image submission over from scratch
First your collections facilitator will completely delete your images from TCIA. Then they will request you to perform these steps before you retransmit your images:
- In the CTP folder, delete these folders: "roots" and "quarantines".
- Kill all java processes.
Fixing a subset of images
If you are only fixing a subset of images you can simply re-send the images. This could include:
- If there was a network problem during transmission and your images were somehow lost in transit (happens very rarely).
- If you realize that you need to modify your de-identification script in some way. In this situation simply make the desired changes to your script and then resend any images which need to be updated.
Transmission errors that can't be resolved by retransmitting the files should be reported to your assigned collections facilitator at UAMS. In an emergency please contact The Cancer Imaging Archive User Support Center @ +1 385-275-8242 (ASK-TCIA).
Sending Log Files
Generally, this will not be required. If your assigned collections facilitator deems it necessary, you will be asked to send the appropriate log files for analysis.
- Log on to the CTP-Client and click Logs.
- Download the log.
Error Reporting HTTP Export Failure
You may encounter an error message suggesting an issue with the HTTP Export process. The message appears as the following:
12:04:49 WARN HttpExportService CT XML HTTP Export: export failed: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target.
Your first response should be to restart the CTP client (refer to Starting CTP Client). If that fails, reboot your computer and then restart the CTP client.
Modifying the CTP Client Configuration
The CTP program uses a configuration file - config.xml. This file is configured for each individual Image Provider site and is automatically installed during the CTP installation process. Should changes be required after the installation of CTP the UAMS collections specialist will provide a revised file. You will need to replace the config.xml file created during the install by following these steps.
- Locate the existing config.xml file in the C:\CTP Client\CTP folder.
- Copy the config.xml file provided by the facilitator to the C:\CTP Client\CTP folder.
Modifying the CTP Client Anonymizer Script
The CTP program requires a customized anonymizer file for each collection. This file is configured for each individual Image Provider site and is automatically installed during the CTP installation process. Should changes be required after the installation of CTP the assigned collections facilitator will provide a revised file. You will need to replace the "project anonymizer".script file created during the install by following these steps.
- Locate the existing "project anonymizer".script in the C:\CTP Client\CTP\scripts folder.
- Copy the "project anonymizer".script file provided by the submissions specialist to the C:\CTP Client\CTP\scripts folder.