Make DICOM reader read time series from 2D images

[goddard@…]

Jason Rubenstein asked about displaying a DICOM time series consisting of 1000 2D images which represent 25 times, each a 3D image of 40 planes. He provided example data (series0079-Body.zip) to Elaine. It opens in ChimeraX 1.3 with menu File / Open DICOM Folder as if it is a single 3D image of 1000 planes. But by splitting the .dcm files into 25 subdirectories ChimeraX can open as 25 3D images and play through them with the mseries command. I've attached a Python scripted I used to put the planes in subdirectories.

Another issue was that the ChimeraX DICOM reader assigned a bad rotation to each volume that goofed up the orthoplanes view -- making the 3 planes not align with the displayed data. To visualize orthoplanes I set the rotation to no rotation with a Python script (attached).

I am not attaching the data as it may be confidential. Here is his original ChimeraX mailing list question

​https://www.rbvi.ucsf.edu/pipermail/chimerax-users/2022-February/003386.html

[Tom Goddard]

Python script to split example data 1000 .dcm files into 25 subdirectories.

[Tom Goddard]

ChimeraX script to set rotation of all volumes to no rotation.

[Zach Pearson]

As I understand, our goal here is to extend the dicom bundle to handle 4D data (in this case, 3D images across 25 time steps).

The heuristic Tom attached to group the time series by filename is good, but may fall apart (as I'm sure he's already aware) if the filename format changes, so the challenge is to incorporate this into dicom.open_dicom cleanly. Today I'm looking for a pattern in the metadata of each file to see if we can divine its rightful place that way.

Should we also incorporate automatically setting all DICOM models to no rotation in the bundle? From a naive perspective I can see no reason a user wouldn't want this, but I'm open to hearing other perspectives.

[Tom Goddard]

Jason says he is happy to provide data and would like to see the images in VR.

Begin forwarded message:

From: "Rubenstein, Jason"
Subject: Re: [chimerax-users] Stack thickness question and cine playing?
Date: March 2, 2022 at 4:53:50 AM PST
To: Tom Goddard

I'd be happy to collaborate! I'd love to get various forms of medical DICOM data into VR. I can certainly provide you with a variety of formats from different imaging modalities.

Jason Rubenstein, M.D., F.A.C.C., F.H.R.S.
Vice Chief, Electrophysiology
Codirector, Cardiac MRI
Associate Professor, Medical College of Wisconsin
Office: 414-955-6777

[Tom Goddard]

I'm going to change Jason to not be the owner of this ticket so he does not have to get spammed with all our technical discussions.

[Zach Pearson]

I added a parameter to open_dicom, 'group_by', which can be used to group DICOM objects by arbitrary metadata. Previously, the DICOM bundle would always assume the images should be grouped by SeriesInstanceUID (which has been set as the default way to open files when group_by is not specified).

I see that when I open files this way different series show up in different colors but I'm betting users would like a way to manipulate all of the images' colors and contrast as if they were part of one series.

[goddard@…]

The DICOM files almost surely contain metadata that describes their grouping.  So no extra argument to open_dicom() should be needed.  The grouping  needs to be inferred from the metadata in the files.

[Zach Pearson]

Yes. In the case of this series of files, the grouping is described by TriggerTime. It's the only field that is the same for every slice in one time step but not the same across all images or unique across all images.

We could narrow the parameter to 'time_series t' and always group by TriggerTime?

[goddard@…]

When opening DICOM, the user should not specify any parameters about what is in the file.  The DICOM standard defines what is in the file in the metadata.  So the reader needs to infer everything from the file.  A user will have no idea how the data is organized — to them it is just a directory with lots of files.

[Zach Pearson]

I don't see how you could better than heuristically approach this. There are a couple hundred metadata tags that may or may not be present:

​https://www.dicomlibrary.com/dicom/dicom-tags/

What's worse is that some of the tags may be private. Any solution that involves us guessing what tag we should group by with no user input is necessarily fragile.

[goddard@…]

The DICOM format defines the contents of the file.  The doctor opening such a dataset in any DICOM software simply opens the file without knowing any parameters.  This is a clinical medicine file format, not some half-baked academic science format that leaves out half the key meta-data values.  So the goal is that ChimeraX interprets the metadata.  I think you got misled by the hack solution of grouping files — that was just because ChiimeraX was too dumb to look at the needed metadata fields.

[Zach Pearson]

I see what you mean. I'll take a deeper dive into what DICOM metadata is always present, and the features of this set's metadata and the new set's metadata that can be used to automatically sort files into their rightful places.

[goddard@…]

Sounds good.  I wrote our original DICOM reading code using the pydicom package.  There are thousands of metadata fields and what you will actually find in the DICOM files is very diverse.  The DICOM specification is thousands of pages.  So we won’t be able to handle much of it.  But our goal will be to handle whatever metadata is needed to correctly open the files of the researchers we are collaborating with.  They may give us very obscure formats that won’t be useful to other users, so it will be good to assess that situation since we want to add capabilities that will be usable by as many researchers as possible.

Maybe Elaine could recommend some free DICOM viewer programs she used, and you can see what they do when opening Jason Rubenstein’s data sets, and they have meta data viewers so you can get a look at what metadata those programs extracted from the file.

Also perhaps Elaine has some DICOM datasets on plato that open correctly in ChimeraX and it would be worth opening those and seeing what ChimeraX does with them (e.g. creates a volume series).

[Zach Pearson]

I first used MatLab just to look at the metadata, then moved on to OsiriX Lite free to see if it would show more detailed metadata (it did not) and if it would open the images correctly (it also did not). With this viewer, it seemed to recognize that there were 25 "albums" somehow, and gave me the option to view it as one 3D series of 1000 images or a 4D set of 25 series of 40 images. Choosing a 4D view, though, it seems to still want to go through the 40 images of one series rather than through one slice in each time step (I apologize if I've confused any terms in what I said, I'll get a clearer view of what they all are but I hope my meaning has come across).

I think we can do better with one of either the TriggerTime or SliceLocation metadata fields.

[Elaine Meng]

See our DICOM index
https://www.rbvi.ucsf.edu/chimerax/dicom/index.html

...which includes a link to my fairly extensive notes on free viewers that I tried, namely Horos, 3D Slicer, and MRIcro.  I don't remember about MRIcro but the first two definitely do a good job at showing metadata. 
https://www.cgl.ucsf.edu/home/meng/dicom/viewers.html

I already sent you two mail about datasets on wynton including ones that (previously anyway) were recognized correctly as time series when opened in ChimeraX, but I'll repeat that here:

[Elaine Meng]

hrm, previous comment seemed to omit the forwarded message within, so I'll try again without the quote-level formatting:

From: Elaine Meng <meng@cgl.ucsf.edu>
Subject: Re: [ChimeraX] #6255: Make DICOM reader read time series from 2D images
Date: March 2, 2022 at 12:41:14 PM PST
To: Zach Pearson <zjp@cgl.ucsf.edu>, Tom Goddard <goddard@cgl.ucsf.edu>

Just wanted to mention that we do already have some examples of DICOM time series that are read in correctly.  So want to make sure that those don't "break" with any of these changes.  
[... edited ...]
As mentioned in this post
https://www.rbvi.ucsf.edu/pipermail/chimerax-users/2022-February/003392.html>

...the time series examples are:

(1) Mouse-Astrocytoma collection, patient ID TVD_GBM_IC1_070610_16_100110, study date 2010-10-01
https://wiki.cancerimagingarchive.net/display/Public/Mouse-Astrocytoma

(2) time series example on 3D slicer wiki
https://wiki.slicer.org/slicerWiki/images/c/c2/DCE_series.zip

I have sample DICOM files in
/wynton/group/ferrin/usr.local/projects/chimerax/www/data/dicom/  ... see the README for list & explanation

for example, #1 above is in file 10-01-2010-339068828-GBMintracranial-631.2.zip

[Elaine Meng]

3D Slicer findings:  I could not figure out how to open a single .dcm alone, so I couldn't open the data that was just sent. 3D Slicer expects a folder, so I tried on the previously sent data, many .dcm files within folder series0079-Body.  Here are screenshots of the resulting appearance of
(1) 3D Slicer DICOM Browser (the one that was just loaded is 4DFlow at the top, ignore the non-highlighted rows which are previous datasets I'd opened in that program)
(2) the additional window I get from clicking the "Metadata" button at the bottom of the DICOM Browser window

[Elaine Meng]

Added by email2trac

[Elaine Meng]

Added by email2trac

[Elaine Meng]

Note that Jason's files are scrubbed of some of the metadata, so what we can see is relatively sparse and perhaps some of the metadata we need to correctly interpret the image data may have been removed... my other examples show much more metadata in these other programs' interfaces, so you really should try them (including the other time series examples) to get a better feel for what is possible.  Also keep in mind that I can barely use these programs, so much of my lack of my successful viewing may be my own ignorance as to how to use the software properly.

Perhaps if he chose different options when exporting the data it would be more interpretable by ChimeraX as well as Horos, 3DSlicer, et al.

Horos findings:  I tried opening the single file we just got and didn't see anything interesting, just that it thought there were 19 images just like ChimeraX did, but I couldn't even figure out how to display any of it.  Patient name is No Name.  See screenshot horos-4D_Ultrasound.png 

I tried opening the 1000-file set from before (File... Import and then choosing all 1000 files in the file browser) and it was more interesting.  See screenshot horos-series0079-Body.png.  If I check the "play" button near the bottom of the dialog it goes through the slices.  However, it may have the same problem as ChimeraX in thinking that it is one big stack rather than divided up into timepoints.

[Elaine Meng]

Added by email2trac

[Elaine Meng]

Added by email2trac

[Zach Pearson]

I will definitely look at the DICOM files that we already have, as well as your notes in depth. I'm operating under the assumption that because we use a folder-based method for opening DICOMs we get lucky when they're well-formed. That seemed to be what Tom was saying, at least I interpreted it to apply to ChimeraX as a whole and not just his script for sorting the files.

It could be good that the series is scrubbed of some metadata; if whatever happens to be left is required for that type of DICOM file it can be relied on to open them. I wanted to put that burden on the user at first; I'm in agreement that that burden shouldn't be there. If the user can specify metadata to sort by, we can probably infer it from the files somehow.

​https://dicom.innolitics.com/ciods

This site lists attributes that may be in each kind of DICOM file that it has listed (many but possibly not all of them). If you click on an attribute it shows whether it is required to be present or not.

[Zach Pearson]

Successfully opened Jason's test dataset using the TriggerTime field to differentiate between time steps. Now the series is opened as 25 different series --- but problems remain. I'm thinking that for DICOM files each volume should be opened in black and white as opposed to each series getting its own tint. That would make it significantly less tedious to set up mseries videos.

Each series' currently displayed frame needs to be changed by hand, but should probably make it so they can be changed together.

Lastly, currently this requires being pretty verbose with mseries slider #1.1.1, could possibly just be mseries slider #1.

[Tom Goddard]

Opening a dicom series should produce a volume series (MapSeries model), not a bunch of separate volumes. A volume series only displays one model, has a single color, and shows a slider for playing through it, related to the "vseries play" command.

[Elaine Meng]

Tom G beat me to the punch but I was going to say much the same thing: should be one mseries made up of 25 3D volumes (each one a time step).   The terminology may be confusing, however, because as I understand it, in DICOM parlance, one series = one 3D volume.

[Zach Pearson]

Sidebar after opening with series grouped by TriggerTime

[Zach Pearson]

I added a screenshot in case I was confusing terms. One model is opened, but the 25 time steps are a couple levels deep and all different in the Volume Viewer.

[goddard@…]

A DICOM time series should open as a volume series controllable with the vseries command.  The mseries command operates on any set of models allowing you to play through them, and is intended primarily for non-volume models.

[goddard@…]

A volume series will only show a single model in the Volume Viewer.  If you are seeing 25 different models then they are not a volume series (ie MapSeries instance).  A volume series makes it easy to adjust the color, threshold level, display style, subsampling... of all time point volumes.

[Zach Pearson]

Here's where I'm confused:

Our Models GUI could imply either one model or 25. The series has ID 1, then Patient has ID 1.1, the date is given ID 1.1.1, and then each time step has its own 1.1.1.{1...25}

I'm having trouble imagining an extension to that volume series logic to the 4th dimension, but I could imagine a Volume Viewer panel with two sliders for time and plane, where the Time slider could be present for any volume but is usually hidden unless we infer we're dealing with a time series.

Would it be correct to say that I need to work on getting this to open with the logic implemented in dicom_grid.py?

[Tom Goddard]

ChimeraX supports a tree of models, so #1 is a model which can have child models such as #1.1, and #1.1 is also a model, and so is #1.1.1. So each node in the tree is a model and can have child models. A MapSeries model has children which are Volume models. The MapSeries model controls the display of its children, for example setting the displayed subregion, threshold, color. In the dicom bundle the dicom_grid.py code assigns each grid a "time" in the DicomGrid constructor. When the data is opened the DicomGrid instances which represent the data without any rendering capability are used to create Volume instances which render the DicomGrid, and if there are multiple times a MapSeries instance is made. So the key is that the "time" of the DicomGrid gets assigned so a MapSeries is created.

[Zach Pearson]

OK, I think I see what you're getting at, but let me check my understanding. In the ChimeraX Python shell I'm looking at the session object and I see

In [16]: session.models.__dict__
Out[16]: 
{'_session': <weakref at 0x7fce50108ea0; to 'Session' at 0x7fce605175e0>,
 '_models': {(1,): <chimerax.core.models.Model at 0x7fce59313310>,
  (1, 1): <chimerax.core.models.Model at 0x7fce8090cbb0>,
  (1, 1, 1): <chimerax.core.models.Model at 0x7fce8090cdc0>,
  (1, 1, 1, 1): <chimerax.map.volume.Volume at 0x7fce62cb7940>,
  (1, 1, 1, 1, 1): <chimerax.map.volume.VolumeImage at 0x7fce806ffd90>,
  (1, 1, 1, 2): <chimerax.map.volume.Volume at 0x7fce808e4250>,
  (1, 1, 1, 3): <chimerax.map.volume.Volume at 0x7fce808e4dc0>,
  (1, 1, 1, 4): <chimerax.map.volume.Volume at 0x7fce808e4a90>,
  (1, 1, 1, 5): <chimerax.map.volume.Volume at 0x7fce808e4760>,
  (1, 1, 1, 6): <chimerax.map.volume.Volume at 0x7fce808e4490>,
  (1, 1, 1, 7): <chimerax.map.volume.Volume at 0x7fce808f6160>,
  (1, 1, 1, 8): <chimerax.map.volume.Volume at 0x7fce808f6460>,
  (1, 1, 1, 9): <chimerax.map.volume.Volume at 0x7fce808f6760>,
  (1, 1, 1, 10): <chimerax.map.volume.Volume at 0x7fce808f6a60>,
  (1, 1, 1, 11): <chimerax.map.volume.Volume at 0x7fce808f6d60>,
  (1, 1, 1, 12): <chimerax.map.volume.Volume at 0x7fce808fd0a0>,
  (1, 1, 1, 13): <chimerax.map.volume.Volume at 0x7fce808fd3a0>,
  (1, 1, 1, 14): <chimerax.map.volume.Volume at 0x7fce808fd6a0>,
  (1, 1, 1, 15): <chimerax.map.volume.Volume at 0x7fce808fd9a0>,
  (1, 1, 1, 16): <chimerax.map.volume.Volume at 0x7fce808fdca0>,
  (1, 1, 1, 17): <chimerax.map.volume.Volume at 0x7fce808fdfa0>,
  (1, 1, 1, 18): <chimerax.map.volume.Volume at 0x7fce809042e0>,
  (1, 1, 1, 19): <chimerax.map.volume.Volume at 0x7fce809045e0>,
  (1, 1, 1, 20): <chimerax.map.volume.Volume at 0x7fce809048e0>,
  (1, 1, 1, 21): <chimerax.map.volume.Volume at 0x7fce80904be0>,
  (1, 1, 1, 22): <chimerax.map.volume.Volume at 0x7fce80904ee0>,
  (1, 1, 1, 23): <chimerax.map.volume.Volume at 0x7fce8090c220>,
  (1, 1, 1, 24): <chimerax.map.volume.Volume at 0x7fce8090c520>,
  (1, 1, 1, 25): <chimerax.map.volume.Volume at 0x7fce8090c820>},
 '_scene_root_model': <chimerax.core.models.Model at 0x7fce4017dd30>,
 '_initialize_camera': False}

Each of 1.1.1.{1...25} will have its own VolumeImage of course (I think this dict only lists the visible ones) but that's beside the point. I think this data structure doesn't represent what we want. Does it make sense to add a representation to chimerax.map.volume that models time, then assign all of the data that's getting split off to different chimerax.map.volume.Volume objects to one chimerax.map.volume.Volume object, letting which data set is displayed change with some chosen time value?

[Tom Goddard]

Volume series have long been supported in ChimeraX. Comment 27 explains what is needed. Setting the time when creating the DicomGrid instances.

[Elaine Meng]

Comment #16 lists sample DICOM time series already correctly shown by ChimeraX, so just open one of those to see how it's supposed to turn out.

[Zach Pearson]

Thanks for the patience, both of you. I think I've got enough info to make headway for now.

[Zach Pearson]

OK, I see how this time logic works and it's just a matter of inferring TemporalPositionIdentifier from TriggerTime.

[Zach Pearson]

​This commit should handle Jason's original dataset. Now when NumberOfTemporalPositions doesn't exist, instead of assuming it's 1 we attempt to infer that data from another field, TriggerTime first. I also added a field to the SeriesFile class to record inferred data fields, so that when we eventually create a DICOM metadata viewer we can make it clear to users that some of the metadata was inferred.

[Zach Pearson]

This series has been readable for months; while it broke for a little while, it's restored in ​this commit for the daily build.