The Cancer Imaging Archive (TCIA) staff has accumulated a wealth of knowledge on best practices and procedures for DICOM image de-identification in the process of maintaining our archive. In order to share this information with the wider research community we are maintaining the following knowledge base. This is a living document and will continue to be updated as we learn from our experiences. If you have feedback or questions please contact us at email@example.com.
Here are some presentations and papers which provide an overview on various aspects of DICOM de-identification and the official Supplement 142 de-identification standards:
- Image Data Sharing for Biomedical Research: Meeting the De-identification and Informatics Challenges publication, Journal of Digital Imaging (DOI: 10.1007/s10278-011-9422-x)
- Image Data Sharing for Biomedical Research: Meeting the De-identification and Informatics Challenges presentation, SIIM Annual Meeting, Washington, D.C., June 4, 2011
- De-identification Revisited - DICOM Supplement 142 presentation, DICOM Conference 2010
- Automated Standards-based Anonymization Profile for Image Sharing Using RSNA's Clinical Trial Processor poster with Q&A session, RSNA Annual Meeting, Chicago, IL, Nov 30, 2009
DICOM Private Data Elements
It is desirable to retain DICOM private data elements that contain parameters describing the acquisition while removing elements containing PHI. Performing this task requires understanding the mechanism defined by DICOM to support private elements. DICOM PS 3.5, section 7.8.1 states:
It is possible that multiple implementors may define Private Elements with the same (odd) group number. To avoid conflicts, Private Elements shall be assigned Private Data Element Tags according to the following rules.
a) Private Creator Data Elements numbered (gggg,0010-00FF) (gggg is odd) shall be used to reserve a block of Elements with Group Number gggg for use by an individual implementor. The implementor shall insert an identification code in the first unused (unassigned) Element in this series to reserve a block of Private Elements. The VR of the private identification code shall be LO (Long String) and the VM shall be equal to 1.
b) Private Creator Data Element (gggg,0010), is a Type 1 Data Element that identifies the implementor reserving element (gggg,1000-10FF), Private Creator Data Element (gggg,0011) identifies the implementor reserving elements (gggg,1100-11FF), and so on, until Private Creator Data Element (gggg,00FF) identifies the implementor reserving elements (gggg,FF00-FFFF).
c) Encoders of Private Data Elements shall be able to dynamically assign private data to any available (unreserved) block(s) within the Private group, and specify this assignment through the blocks corresponding Private Creator Data Element(s). Decoders of Private Data shall be able to accept reserved blocks with a given Private Creator identification code at any position within the Private group specified by the blocks corresponding Private Creator Data Element.
We will use data in group 0009 as a practical example. The table below shows an example of data that could be included in group 0009.
Private Creator Element
Density Standard Deviation
In the example, the element with tag (0009, 0010) is a private creator element with value "ACME". That reserves a block of elements for this manufacturer. The element (0009, 1001) is part of that block; the 10 in the element tag (1001) corresponds to the 10 that is in the tag of the Private Creator Element (0009, 0010).
This only becomes complex when different manufacturers want to use the same reserved block to store information. When this occurs in a single image, the creator of the image reserves a block (for example, 0010). When a second application wants to add data to that same group, it detects the block written by the creator and creates a separate block (for example, 0011). The creator is not required to start at block 0010, but that appears to be common practice. The second or third application is not required to use 0011 or 0012. Based on this encoding scheme, some observations are:
- If a collection of images are produced by equipment from different manufacturers, you may have collisions in the sets of private elements you want to retain and discard. For example, element (0009, 1001) from manufacturer A may contain an important physical parameter while that same element from manufacturer B may contain PHI.
- If the collection has images that are created by an acquisition modality and are then modified by another application (PACS, workstation), a private group may have multiple reserved blocks. Also, one cannot assume that the original creator will have always chosen reserved block 0010.
DICOM Basic Attribute Confidentiality Profile
DICOM standards committee Working Group 18 wrote Supplement 142 that is now incorporated into the published DICOM Standard. The Attribute Confidentiality Profile (DICOM PS 3.15: Appendix E) provides a standard for image de-identification and a process with which to reduce the complexity involved in safely de‐identifying DICOM image data while providing flexibility for scenarios which necessitate preservation of certain information needed for quality control and analysis that is essential to research. This is achieved by providing a number of Application Level Confidentiality Profiles which includes a Basic Profile along with a number of Option Profiles. These profiles provide the necessary instructions for how to safely clean DICOM elements which may contain PHI. The DICOM Standard, including Part 15, is available at the NEMA web site: http://medical.nema.org/standard.html The original Supplement 142 guidance document can be obtained at ftp://medical.nema.org/medical/dicom/final/sup142_ft.doc. We recommend you use the published standard above as it will be updated with any change proposals.
Appendix E of PS 3.15 documents a system for protecting attributes. We quote a small section of the document.
The Attributes listed in Table E.1-1 for each profile are contained in Standard IODs, or may be contained in Standard Extended IODs. An implementation claiming conformance to an Application Level
Confidentiality Profile as a de-identifier shall protect or retain all instances of the Attributes listed in Table E.1-1, whether contained in the main dataset or embedded in an Item of a Sequence of Items. The following action codes are used in the table:
– D – replace with a non-zero length value that may be a dummy value and consistent with the VR
– Z – replace with a zero length value, or a non-zero length value that may be a dummy value and consistent with the VR
– X – remove
– K – keep (unchanged for non-sequence attributes, cleaned for sequences)
– C – clean, that is replace with values of similar meaning known not to contain identifying information and consistent with the VR
– U – replace with a non-zero length UID that is internally consistent within a set of Instances
– Z/D – Z unless D is required to maintain IOD conformance (Type 2 versus Type 1)
– X/Z – X unless Z is required to maintain IOD conformance (Type 3 versus Type 2)
– X/D – X unless D is required to maintain IOD conformance (Type 3 versus Type 1)
– X/Z/D – X unless Z or D is required to maintain IOD conformance (Type 3 versus Type 2 versus Type 1)
– X/Z/U* - X unless Z or replacement of contained instance UIDs (U) is required to maintain IOD conformance (Type 3 versus Type 2 versus Type 1 sequences containing UID references)
PS 3.15: E.2 then defines the Basic Application Level Confidentiality Profile which describes how to apply the scheme above with a number of options that determine the scope of protection that is provided. These definitions allow a system to follow a standard procedure and document in a standard way the behavior of that system.
- Download the software
- Read the documentation
- Join the User Group mailing list
- Extending/Contributing CTP source code
TCIA utilizes the RSNA Clinical Trials Processor (CTP) software in conjunction with caBIG's National Biomedical Imaging Archive (NBIA) to de‐identify and host the images in the archive. The Cancer Imaging Program's Informatics Team has been working closely with the developer of CTP since 2009 to incorporate support for this standard as it was being defined by WG18. A full summary and time line of this project can be found athttps://wiki.nci.nih.gov/display/CIP/Incorporation+of+DICOM+WG18+Supplement+142+into+CTP.
CTP provides an interface that allows application of any combination of the profiles to a set of images, and allows for application of an audit trail for retroactively tracking applied de‐identification. For images that are submitted to TCIA the staff begins with the Basic Application Confidentiality Profile (which is the most aggressive) in combination with the following options:
- Clean Descriptors Option: Removal of identification information from descriptive tags which contain unstructured plain text values over which an operator has control
- Retain Modified Longitudinal Temporal Information Options: Modification of tags that contain dates or times
- Retain Patient Characteristics Option: Retention of physical characteristics of the patient that are descriptive rather than identifying information (e.g. metabolic measures, body weight, etc.)
- Retain Device Identity Option: Retention of information about the characteristics of the device used to perform the acquisition
- Retain Safe Private Option: Retention of Private Attributes confirmed not to contain PHI
DICOM Tag Sniffer
In order to simplify our ability to implement some of the "clean" instructions specified in DICOM PS 3.15 a new tool was developed to help inspect the contents of DICOM elements which allow free text entry by a technician and Private Tags for potential PHI. This tool scans a folder and included subfolders for DICOM objects and produces several different outputs that depend on the mode used and input profiles. The software reads each DICOM object and iterates through each public and private element. The software then uses the profiles below to determine whether to retain the value of the element for later inspection:
- Confidentiality Profile: One input profile corresponds to the entries in table E.1-1 in DICOM PS 3.15. We list the attributes in the table and the coded values according to the table entries. When scanning the DICOM objects, each public element is checked against the data in the profile. If the element is found in the profile, the software knows if it should record the element value for later inspection or if the software can ignore it. For example, if the DICOM profile indicates the element is to be deleted, there is no reason to review the value in that element.
- The Confidentiality Profile input is augmented with elements that are known to contain physical parameters such as rows, columns or pixel spacing. Rather than tell the software to ignore values with a specific value representation, we list those elements explicitly.
- Modality Software Profile: This input profile describes the private elements that are documented in the conformance statement by the manufacturer. This file takes into account the Private Creator Data Elements described above and has a code table for indicating program actions (record the value, ignore the value, ...)
These outputs are relevant at different stages of the curation and image publication process.
- Element Inventory: is the set of DICOM tags that are found in the image set. The tags include only the hexadecimal tags (xxxx, yyyy) and no values. All public and private tags are listed, but each is listed only once. The Confidentiality Profile and Modality Software Profile are not consulted as no values are retained for review.
- Element Values, Pre-Deidentification: We want to examine element values to determine how to configure CTP scripts for proper de-identification. As mentioned above, we want to retain as many elements as possible while not exposing PHI. We also do not want to review all element values in all DICOM objects. We use a Confidentiality Profile that corresponds to the DICOM Basic Application Confidentiality Profile and a Modality Software Profile that properly describes the private elements in the DICOM objects.
- Element Values, Final Review: In this mode, we want to review the values in the DICOM objects just before publication. We have de-identified the data and want to analyze the data as a final check. In this mode, we use a different Confidentiality Profile and different Modality Software Profile. For the Confidentiality Profile, we only list elements that we know are physical parameters (rows, columns, ....) and do not include the DICOM references from PS 3.15, Table E.1-1. That will direct the software to record the element values. Likewise, the Modality Software Profile used will direct the software to record all values for later analysis.
We believe this tool might be useful to the rest of the research community and so it's been made freely available as an open source application. We have also created documentation for how a researcher could utilize in the context of their own projects.
TCIA De-identification Work Flow
The TCIA provides standards‐based curation support to ensure safe and thorough de‐identification of all images in the archive per federal HIPAA and HITECH regulations. In order to achieve this compliance without stripping the data of its scientific utility TCIA staff perform a redundant, thorough de‐identification and analysis procedure based on guidance provided by the industry experts in DICOM standards committee Working Group 18. Each collection submitted for publication is analyzed and de-identified as a whole using the steps listed below. All steps are completed before the collection is released for publication.
- Each image in the collection is visually inspected to guarantee there is no PHI burned into the pixel data.
- TagSniffer is used to review the collection and produce an Element Inventory that is annotated with data from the DICOM Basic Application Confidentiality Profile and our set of Modality Software Profiles. This produces the list of DICOM elements found in the collection with a simple annotation scheme:
- One of the Basic Application Confidentiality Profile codes that indicates the DICOM scheme for de-identification (if the element is listed by DICOM)
- A simple code from our Modality Software Profile (No PHI: Retain, PHI: Delete, Not Sure: Review)
- No code, indicating the element is not registered
- The Pre-Identification output of the Tag Sniffer is also generated. This will contain the set of elements in the collection and all values that need to be reviewed for PHI. If the Basic Application Confidentiality Profile or applicable Modality Software Profile indicates the attribute is to be cleaned or that the attribute is a physical parameter that does not contain PHI, there is no need to review that element at this step. We know that our de-identification script will process the element properly.
- We combine the information from steps 2 and 3 to create a CTP de-identification script for the collection. In the event of multiple scanners from different manufacturers, we might create and apply different scripts based on manufacturer.
- The CTP de-identification script (or scripts) is (are) applied to the image collection and a separate copy of the images is created. That is, we retain the original set in case we need to repeat a step.
- TagSniffer is used to review the de-identified images and create the Final Review Output. This is a more complete output that is reviewed by analysts to guarantee there is no PHI carried forward after de-identification. Both public and private elements are included in the output for review.
- If any errors are detected in de-identification in step 6, the CTP script is adjusted and the image set is processed again starting at step 5.
Only after this inspection is complete are the images made available to the general public. For general information on what to expect as an image provider please see our web site at http://www.cancerimagingarchive.net/provider.html.
Manufacturer Specific Private Tags
As discussed above, medical manufacturers include private elements in their DICOM images to convey information not defined in the DICOM Standard. This section documents the information we have gathered by reading appropriate conformance statements.
The sections below describe information by manufacturer. That information is encoded in files that describe the private elements created by those manufacturers. Those files are part of the run time environment of the Tag Sniffer and are maintained in our forge: https://mirgforge.wustl.edu/gf/project/dicomtagsniffer/scmsvn/?action=browse&path=%2Ftrunk%2Fdeploy%2Fprofiles%2Fdevice-profiles%2F