Opened 7 years ago

Closed 3 years ago

#1745 closed enhancement (wontfix)

Error opening DICOM with lossless JPEG image encoding

Reported by: goddard@… Owned by: Zach Pearson
Priority: normal Milestone:
Component: DICOM Version:
Keywords: Cc: pett, Greg Couch
Blocked By: Blocking:
Notify when closed: Platform: all
Project: ChimeraX

Description 

The following bug report has been submitted:
Platform:        Darwin-17.7.0-x86_64-i386-64bit
ChimeraX Version: 0.9 (2019-02-19)
Description
Opening the attached DICOM file with lossless JPEG image encoding gives this error.

Log:
UCSF ChimeraX version: 0.9 (2019-02-19)  
© 2016-2019 Regents of the University of California. All rights reserved.  
How to cite UCSF ChimeraX  

> open
/Users/goddard/ucsf/data/dicom/02ef8f31ea86a45cfce6eb297c274598/series-000001/image-000001.dcm

Traceback (most recent call last):  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/pydicom/pixel_data_handlers/pillow_handler.py", line 197, in
get_pixeldata  
decompressed_image = Image.open(fio)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/PIL/Image.py", line 2687, in open  
% (filename if filename else fp))  
OSError: cannot identify image file <_io.BytesIO object at 0x13d5254c0>  
  
During handling of the above exception, another exception occurred:  
  
Traceback (most recent call last):  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/cmd_line/tool.py", line 252, in execute  
cmd.run(cmd_text)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/core/commands/cli.py", line 2617, in run  
result = ci.function(session, **kw_args)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/core/commands/open.py", line 41, in open  
from_database=from_database, ignore_cache=ignore_cache, **kw))  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/core/commands/open.py", line 117, in open  
models = handle_unknown_kw(session.models.open, paths, format=format,
name=name, **kw)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/core/commands/open.py", line 62, in handle_unknown_kw  
return f(*args, **kw)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/core/models.py", line 598, in open  
session, filenames, format=format, name=name, **kw)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/core/io.py", line 463, in open_multiple_data  
models, status = open_func(session, paths, mname, **kw)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/dicom/dicom.py", line 39, in open_dicom  
models, msg = dicom_volumes(session, image_series, **kw)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/dicom/dicom.py", line 117, in dicom_volumes  
gmodels, gmsg = open_grids(session, grid_group, gname, **kw)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/map/volume.py", line 3222, in open_grids  
v.initialize_thresholds()  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/map/volume.py", line 518, in initialize_thresholds  
s = self.matrix_value_statistics()  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/map/volume.py", line 1613, in matrix_value_statistics  
matrices = self.displayed_matrices(read_matrix)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/map/volume.py", line 1648, in displayed_matrices  
matrices.append(self.matrix(read_matrix))  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/map/volume.py", line 1163, in matrix  
m = self.region_matrix(r, read_matrix)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/map/volume.py", line 1187, in region_matrix  
m = d.matrix(origin, size, step, progress, from_cache_only)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/map/data/griddata.py", line 249, in matrix  
m = self._read_full_planes(ijk_origin, ijk_size, ijk_step, progress)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/map/data/griddata.py", line 274, in _read_full_planes  
fm = self.read_matrix(f_ijk_origin, f_ijk_size, f_ijk_step, progress)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/dicom/dicom_grid.py", line 117, in read_matrix  
self.time, c, m, progress)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/dicom/dicom_format.py", line 523, in read_matrix  
p = self.read_plane(k, time, channel)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/dicom/dicom_format.py", line 541, in read_plane  
data = d.pixel_array  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/pydicom/dataset.py", line 929, in pixel_array  
self.convert_pixel_data()  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/pydicom/dataset.py", line 875, in convert_pixel_data  
raise last_exception  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/pydicom/dataset.py", line 843, in convert_pixel_data  
arr = handler.get_pixeldata(self)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/pydicom/pixel_data_handlers/pillow_handler.py", line 199, in
get_pixeldata  
raise NotImplementedError(e.strerror)  
NotImplementedError: None  
  
NotImplementedError: None  
  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/pydicom/pixel_data_handlers/pillow_handler.py", line 199, in
get_pixeldata  
raise NotImplementedError(e.strerror)  
  
See log for complete Python traceback.  
  
Traceback (most recent call last):  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/pydicom/pixel_data_handlers/pillow_handler.py", line 197, in
get_pixeldata  
decompressed_image = Image.open(fio)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/PIL/Image.py", line 2687, in open  
% (filename if filename else fp))  
OSError: cannot identify image file <_io.BytesIO object at 0x13d5c0990>  
  
During handling of the above exception, another exception occurred:  
  
Traceback (most recent call last):  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/core/triggerset.py", line 130, in invoke  
return self._func(self._name, data)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/map/volume.py", line 3470, in _update_drawings  
v.initialize_thresholds()  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/map/volume.py", line 518, in initialize_thresholds  
s = self.matrix_value_statistics()  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/map/volume.py", line 1613, in matrix_value_statistics  
matrices = self.displayed_matrices(read_matrix)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/map/volume.py", line 1648, in displayed_matrices  
matrices.append(self.matrix(read_matrix))  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/map/volume.py", line 1163, in matrix  
m = self.region_matrix(r, read_matrix)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/map/volume.py", line 1187, in region_matrix  
m = d.matrix(origin, size, step, progress, from_cache_only)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/map/data/griddata.py", line 249, in matrix  
m = self._read_full_planes(ijk_origin, ijk_size, ijk_step, progress)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/map/data/griddata.py", line 274, in _read_full_planes  
fm = self.read_matrix(f_ijk_origin, f_ijk_size, f_ijk_step, progress)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/dicom/dicom_grid.py", line 117, in read_matrix  
self.time, c, m, progress)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/dicom/dicom_format.py", line 523, in read_matrix  
p = self.read_plane(k, time, channel)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/chimerax/dicom/dicom_format.py", line 541, in read_plane  
data = d.pixel_array  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/pydicom/dataset.py", line 929, in pixel_array  
self.convert_pixel_data()  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/pydicom/dataset.py", line 875, in convert_pixel_data  
raise last_exception  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/pydicom/dataset.py", line 843, in convert_pixel_data  
arr = handler.get_pixeldata(self)  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/pydicom/pixel_data_handlers/pillow_handler.py", line 199, in
get_pixeldata  
raise NotImplementedError(e.strerror)  
NotImplementedError: None  
  
Error processing trigger "graphics update": None:  
NotImplementedError: None  
  
File
"/Users/goddard/Desktop/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-
packages/pydicom/pixel_data_handlers/pillow_handler.py", line 199, in
get_pixeldata  
raise NotImplementedError(e.strerror)  
  
See log for complete Python traceback.  
  




OpenGL version: 4.1 NVIDIA-10.32.0 355.11.10.10.40.102
OpenGL renderer: NVIDIA GeForce GTX 680MX OpenGL Engine
OpenGL vendor: NVIDIA Corporation
File attachment: image-000001.dcm

image-000001.dcm​

Attachments (2)

image-000001.dcm​ (241.0 KB ) - added by goddard@… 7 years ago.
Added by email2trac
Screenshot 2022-06-22 at 17.54.03.png​ (3.7 MB ) - added by Zach Pearson 3 years ago.
I installed python-gdcm==3.0.12 (supports up to Python 3.10) into my ChimeraX on my laptop (arm64 macOS Ventura beta 2) and it opens the attached file with no complaints.

Change History (40)

by goddard@…, 7 years ago

Attachment: image-000001.dcm​ added

Added by email2trac

comment:1 by Tom Goddard, 7 years ago

Component: Unassigned → Volume Data
Owner: set to Tom Goddard
Platform: → all
Project: → ChimeraX
Status: new → assigned
Summary: ChimeraX bug report submission → Error opening DICOM with lossless JPEG image encoding

comment:2 by Tom Goddard, 7 years ago

The error is because PyDICOM thinks Pillow can open this lossless jpeg format image but the pillow jpeg reader raises an error. pydicom masks the actual error that pillow produces. The error from Pillow is:

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/PIL/JpegImagePlugin.py", line 166, in SOF

raise SyntaxError("cannot handle %d-bit layers" % self.bits)

File "<string>", line None

SyntaxError: cannot handle 16-bit layers

comment:3 by Tom Goddard, 7 years ago

Here is more debugging output including DICOM header values. The dicom transfer syntax is

transfer syntax /Users/goddard/ucsf/data/dicom/02ef8f31ea86a45cfce6eb297c274598/series-000001/image-000001.dcm 1.2.840.10008.1.2.4.70

which is lossless jpeg and it is 16 bit.

UCSF ChimeraX version: 0.9 (2019-02-21)
© 2016-2019 Regents of the University of California. All rights reserved.
How to cite UCSF ChimeraX
open /Users/goddard/ucsf/data/dicom/02ef8f31ea86a45cfce6eb297c274598/series-000001/image-000001.dcm
Summary of feedback from opening /Users/goddard/ucsf/data/dicom/02ef8f31ea86a45cfce6eb297c274598/series-000001/image-000001.dcm
notes transfer syntax /Users/goddard/ucsf/data/dicom/02ef8f31ea86a45cfce6eb297c274598/series-000001/image-000001.dcm 1.2.840.10008.1.2.4.70
trying format BMP result False
trying format GIF result False
trying format TIFF result False
trying format JPEG result True
failed decoding pixel data /Users/goddard/ucsf/data/dicom/02ef8f31ea86a45cfce6eb297c274598/series-000001/image-000001.dcm (0008, 0005) Specific Character Set CS: 'ISO_IR 100'
(0008, 0008) Image Type CS: ['ORIGINAL', 'PRIMARY', 'OTHER']
(0008, 0012) Instance Creation Date DA: '20070101'
(0008, 0013) Instance Creation Time TM: '120000.000000'
(0008, 0016) SOP Class UID UI: MR Image Storage
(0008, 0018) SOP Instance UID UI: 1.2.826.0.1.3680043.8.1055.1.20111103111148288.81713267.86905863
(0008, 0020) Study Date DA: '20070101'
(0008, 0021) Series Date DA: '20070101'
(0008, 0022) Acquisition Date DA: '20070101'
(0008, 0023) Content Date DA: '20070101'
(0008, 002a) Acquisition DateTime DT: '20070101120000'
(0008, 0030) Study Time TM: '120000.000000'
(0008, 0031) Series Time TM: '120000.000000'
(0008, 0032) Acquisition Time TM: '120000.000000'
(0008, 0033) Content Time TM: '120000.000000'
(0008, 0060) Modality CS: 'MR'
(0008, 1030) Study Description LO: 'Knee (R)'
(0008, 103e) Series Description LO: 'AX. FSE PD\x00'
(0008, 1140) Referenced Image Sequence 2 item(s) ----
(0008, 1150) Referenced SOP Class UID UI: MR Image Storage
(0008, 1155) Referenced SOP Instance UID UI: 1.2.840.113619.2.176.2025.1499492.7040.1171286241.719

(0008, 1150) Referenced SOP Class UID UI: MR Image Storage
(0008, 1155) Referenced SOP Instance UID UI: 1.2.840.113619.2.176.2025.1499492.7040.1171286241.726

(0008, 2111) Derivation Description ST: 'Lossless JPEG compression, selection value 1, point transform 0, compression ratio 2.1453 [Lossless JPEG compression, selection value 1, point transform 0, compression ratio 2.1453]\x00'
(0008, 9215) Derivation Code Sequence 1 item(s) ----
(0008, 0100) Code Value SH: '121327'
(0008, 0102) Coding Scheme Designator SH: 'DCM\x00'
(0008, 0104) Code Meaning LO: 'Full fidelity image, uncompressed or lossless compressed'

(0010, 0010) Patient's Name PN: 'Anonymized'
(0010, 0020) Patient ID LO: '0\x00'
(0010, 1010) Patient's Age AS: '000Y'
(0010, 1030) Patient's Weight DS: "0"
(0018, 0020) Scanning Sequence CS: 'SE'
(0018, 0021) Sequence Variant CS: ['SK', 'OSP']
(0018, 0022) Scan Options CS: ['SAT_GEMS', 'NPW', 'TRF_GEMS', 'FILTERED_GEMS', 'FS']
(0018, 0023) MR Acquisition Type CS: '2D'
(0018, 0025) Angio Flag CS: 'N'
(0018, 0050) Slice Thickness DS: "4"
(0018, 0080) Repetition Time DS: "2800"
(0018, 0081) Echo Time DS: "27.524"
(0018, 0082) Inversion Time DS: "0"
(0018, 0083) Number of Averages DS: "1"
(0018, 0084) Imaging Frequency DS: "63.860145"
(0018, 0085) Imaged Nucleus SH: '1H'
(0018, 0086) Echo Number(s) IS: "1"
(0018, 0087) Magnetic Field Strength DS: "1.5"
(0018, 0088) Spacing Between Slices DS: "4.5"
(0018, 0091) Echo Train Length IS: "10"
(0018, 0093) Percent Sampling DS: "100"
(0018, 0094) Percent Phase Field of View DS: "100"
(0018, 0095) Pixel Bandwidth DS: "162.773"
(0018, 1030) Protocol Name LO: '324-58-2995/6\x00'
(0018, 1088) Heart Rate IS: "474"
(0018, 1090) Cardiac Number of Images IS: "0"
(0018, 1094) Trigger Window IS: "0"
(0018, 1100) Reconstruction Diameter DS: "170"
(0018, 1250) Receive Coil Name SH: 'HD TRknee PA'
(0018, 1310) Acquisition Matrix US: [0, 384, 224, 0]
(0018, 1312) In-plane Phase Encoding Direction CS: 'ROW'
(0018, 1314) Flip Angle DS: "90"
(0018, 1315) Variable Flip Angle Flag CS: 'N'
(0018, 1316) SAR DS: "0.0533"
(0018, 5100) Patient Position CS: 'FFS'
(0020, 000d) Study Instance UID UI: 1.2.826.0.1.3680043.8.1055.1.20111103111148288.98361414.79379639
(0020, 000e) Series Instance UID UI: 1.2.826.0.1.3680043.8.1055.1.20111103111148288.94019146.71622702
(0020, 0011) Series Number IS: "5"
(0020, 0012) Acquisition Number IS: "1"
(0020, 0013) Instance Number IS: "1"
(0020, 0032) Image Position (Patient) DS: ['-149.033', '-118.499', '-61.0464']
(0020, 0037) Image Orientation (Patient) DS: ['0.999841', '0.000366209', '0.0178227', '-0.000427244', '0.999995', '0.00326545']
(0020, 0052) Frame of Reference UID UI: 1.2.840.113619.2.176.2025.1499492.7391.1171285944.389
(0020, 1002) Images in Acquisition IS: "24"
(0020, 1041) Slice Location DS: "-59.25741196"
(0028, 0002) Samples per Pixel US: 1
(0028, 0004) Photometric Interpretation CS: 'MONOCHROME2'
(0028, 0010) Rows US: 512
(0028, 0011) Columns US: 512
(0028, 0030) Pixel Spacing DS: ['0.332', '0.332']
(0028, 0100) Bits Allocated US: 16
(0028, 0101) Bits Stored US: 16
(0028, 0102) High Bit US: 15
(0028, 0103) Pixel Representation US: 1
(0028, 0106) Smallest Image Pixel Value SS: 0
(0028, 0107) Largest Image Pixel Value SS: 5145
(0028, 0120) Pixel Padding Value SS: 0
(0028, 1050) Window Center DS: "2572"
(0028, 1051) Window Width DS: "5145"
(7fe0, 0010) Pixel Data OB: Array of 244404 bytes
Traceback (most recent call last):

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/cmd_line/tool.py", line 252, in execute

cmd.run(cmd_text)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/core/commands/cli.py", line 2617, in run

result = ci.function(session, kw_args)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/core/commands/open.py", line 41, in open

from_database=from_database, ignore_cache=ignore_cache, kw))

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/core/commands/open.py", line 117, in open

models = handle_unknown_kw(session.models.open, paths, format=format, name=name, kw)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/core/commands/open.py", line 62, in handle_unknown_kw

return f(*args, kw)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/core/models.py", line 598, in open

session, filenames, format=format, name=name, kw)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/core/io.py", line 463, in open_multiple_data

models, status = open_func(session, paths, mname, kw)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/dicom/dicom.py", line 39, in open_dicom

models, msg = dicom_volumes(session, image_series, kw)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/dicom/dicom.py", line 117, in dicom_volumes

gmodels, gmsg = open_grids(session, grid_group, gname, kw)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/map/volume.py", line 3240, in open_grids

v.initialize_thresholds()

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/map/volume.py", line 520, in initialize_thresholds

s = self.matrix_value_statistics()

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/map/volume.py", line 1629, in matrix_value_statistics

matrices = self.displayed_matrices(read_matrix)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/map/volume.py", line 1664, in displayed_matrices

matrices.append(self.matrix(read_matrix))

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/map/volume.py", line 1179, in matrix

m = self.region_matrix(r, read_matrix)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/map/volume.py", line 1203, in region_matrix

m = d.matrix(origin, size, step, progress, from_cache_only)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/map/data/griddata.py", line 249, in matrix

m = self._read_full_planes(ijk_origin, ijk_size, ijk_step, progress)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/map/data/griddata.py", line 274, in _read_full_planes

fm = self.read_matrix(f_ijk_origin, f_ijk_size, f_ijk_step, progress)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/dicom/dicom_grid.py", line 119, in read_matrix

self.time, c, m, progress)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/dicom/dicom_format.py", line 539, in read_matrix

p = self.read_plane(k, time, channel)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/dicom/dicom_format.py", line 558, in read_plane

data = d.pixel_array

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pydicom/dataset.py", line 929, in pixel_array

self.convert_pixel_data()

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pydicom/dataset.py", line 875, in convert_pixel_data

raise last_exception

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pydicom/dataset.py", line 843, in convert_pixel_data

arr = handler.get_pixeldata(self)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pydicom/pixel_data_handlers/pillow_handler.py", line 197, in get_pixeldata

decompressed_image = Image.open(fio)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/PIL/Image.py", line 2675, in open

im = _open_core(fp, filename, prefix)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/PIL/Image.py", line 2659, in _open_core

im = factory(fp, filename)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/PIL/JpegImagePlugin.py", line 779, in jpeg_factory

im = JpegImageFile(fp, filename)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/PIL/ImageFile.py", line 103, in init

self._open()

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/PIL/JpegImagePlugin.py", line 338, in _open

handler(self, i)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/PIL/JpegImagePlugin.py", line 166, in SOF

raise SyntaxError("cannot handle %d-bit layers" % self.bits)

File "<string>", line None

SyntaxError: cannot handle 16-bit layers

File "", line None
SyntaxError: cannot handle 16-bit layers

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/PIL/JpegImagePlugin.py", line 166, in SOF
raise SyntaxError("cannot handle %d-bit layers" % self.bits)

See log for complete Python traceback.

trying format BMP result False
trying format GIF result False
trying format TIFF result False
trying format JPEG result True
failed decoding pixel data /Users/goddard/ucsf/data/dicom/02ef8f31ea86a45cfce6eb297c274598/series-000001/image-000001.dcm (0008, 0005) Specific Character Set CS: 'ISO_IR 100'
(0008, 0008) Image Type CS: ['ORIGINAL', 'PRIMARY', 'OTHER']
(0008, 0012) Instance Creation Date DA: '20070101'
(0008, 0013) Instance Creation Time TM: '120000.000000'
(0008, 0016) SOP Class UID UI: MR Image Storage
(0008, 0018) SOP Instance UID UI: 1.2.826.0.1.3680043.8.1055.1.20111103111148288.81713267.86905863
(0008, 0020) Study Date DA: '20070101'
(0008, 0021) Series Date DA: '20070101'
(0008, 0022) Acquisition Date DA: '20070101'
(0008, 0023) Content Date DA: '20070101'
(0008, 002a) Acquisition DateTime DT: '20070101120000'
(0008, 0030) Study Time TM: '120000.000000'
(0008, 0031) Series Time TM: '120000.000000'
(0008, 0032) Acquisition Time TM: '120000.000000'
(0008, 0033) Content Time TM: '120000.000000'
(0008, 0060) Modality CS: 'MR'
(0008, 1030) Study Description LO: 'Knee (R)'
(0008, 103e) Series Description LO: 'AX. FSE PD\x00'
(0008, 1140) Referenced Image Sequence 2 item(s) ----
(0008, 1150) Referenced SOP Class UID UI: MR Image Storage
(0008, 1155) Referenced SOP Instance UID UI: 1.2.840.113619.2.176.2025.1499492.7040.1171286241.719

(0008, 1150) Referenced SOP Class UID UI: MR Image Storage
(0008, 1155) Referenced SOP Instance UID UI: 1.2.840.113619.2.176.2025.1499492.7040.1171286241.726

(0008, 2111) Derivation Description ST: 'Lossless JPEG compression, selection value 1, point transform 0, compression ratio 2.1453 [Lossless JPEG compression, selection value 1, point transform 0, compression ratio 2.1453]\x00'
(0008, 9215) Derivation Code Sequence 1 item(s) ----
(0008, 0100) Code Value SH: '121327'
(0008, 0102) Coding Scheme Designator SH: 'DCM\x00'
(0008, 0104) Code Meaning LO: 'Full fidelity image, uncompressed or lossless compressed'

(0010, 0010) Patient's Name PN: 'Anonymized'
(0010, 0020) Patient ID LO: '0\x00'
(0010, 1010) Patient's Age AS: '000Y'
(0010, 1030) Patient's Weight DS: "0"
(0018, 0020) Scanning Sequence CS: 'SE'
(0018, 0021) Sequence Variant CS: ['SK', 'OSP']
(0018, 0022) Scan Options CS: ['SAT_GEMS', 'NPW', 'TRF_GEMS', 'FILTERED_GEMS', 'FS']
(0018, 0023) MR Acquisition Type CS: '2D'
(0018, 0025) Angio Flag CS: 'N'
(0018, 0050) Slice Thickness DS: "4"
(0018, 0080) Repetition Time DS: "2800"
(0018, 0081) Echo Time DS: "27.524"
(0018, 0082) Inversion Time DS: "0"
(0018, 0083) Number of Averages DS: "1"
(0018, 0084) Imaging Frequency DS: "63.860145"
(0018, 0085) Imaged Nucleus SH: '1H'
(0018, 0086) Echo Number(s) IS: "1"
(0018, 0087) Magnetic Field Strength DS: "1.5"
(0018, 0088) Spacing Between Slices DS: "4.5"
(0018, 0091) Echo Train Length IS: "10"
(0018, 0093) Percent Sampling DS: "100"
(0018, 0094) Percent Phase Field of View DS: "100"
(0018, 0095) Pixel Bandwidth DS: "162.773"
(0018, 1030) Protocol Name LO: '324-58-2995/6\x00'
(0018, 1088) Heart Rate IS: "474"
(0018, 1090) Cardiac Number of Images IS: "0"
(0018, 1094) Trigger Window IS: "0"
(0018, 1100) Reconstruction Diameter DS: "170"
(0018, 1250) Receive Coil Name SH: 'HD TRknee PA'
(0018, 1310) Acquisition Matrix US: [0, 384, 224, 0]
(0018, 1312) In-plane Phase Encoding Direction CS: 'ROW'
(0018, 1314) Flip Angle DS: "90"
(0018, 1315) Variable Flip Angle Flag CS: 'N'
(0018, 1316) SAR DS: "0.0533"
(0018, 5100) Patient Position CS: 'FFS'
(0020, 000d) Study Instance UID UI: 1.2.826.0.1.3680043.8.1055.1.20111103111148288.98361414.79379639
(0020, 000e) Series Instance UID UI: 1.2.826.0.1.3680043.8.1055.1.20111103111148288.94019146.71622702
(0020, 0011) Series Number IS: "5"
(0020, 0012) Acquisition Number IS: "1"
(0020, 0013) Instance Number IS: "1"
(0020, 0032) Image Position (Patient) DS: ['-149.033', '-118.499', '-61.0464']
(0020, 0037) Image Orientation (Patient) DS: ['0.999841', '0.000366209', '0.0178227', '-0.000427244', '0.999995', '0.00326545']
(0020, 0052) Frame of Reference UID UI: 1.2.840.113619.2.176.2025.1499492.7391.1171285944.389
(0020, 1002) Images in Acquisition IS: "24"
(0020, 1041) Slice Location DS: "-59.25741196"
(0028, 0002) Samples per Pixel US: 1
(0028, 0004) Photometric Interpretation CS: 'MONOCHROME2'
(0028, 0010) Rows US: 512
(0028, 0011) Columns US: 512
(0028, 0030) Pixel Spacing DS: ['0.332', '0.332']
(0028, 0100) Bits Allocated US: 16
(0028, 0101) Bits Stored US: 16
(0028, 0102) High Bit US: 15
(0028, 0103) Pixel Representation US: 1
(0028, 0106) Smallest Image Pixel Value SS: 0
(0028, 0107) Largest Image Pixel Value SS: 5145
(0028, 0120) Pixel Padding Value SS: 0
(0028, 1050) Window Center DS: "2572"
(0028, 1051) Window Width DS: "5145"
(7fe0, 0010) Pixel Data OB: Array of 244404 bytes
Traceback (most recent call last):

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/core/triggerset.py", line 130, in invoke

return self._func(self._name, data)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/map/volume.py", line 3488, in _update_drawings

v.initialize_thresholds()

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/map/volume.py", line 520, in initialize_thresholds

s = self.matrix_value_statistics()

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/map/volume.py", line 1629, in matrix_value_statistics

matrices = self.displayed_matrices(read_matrix)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/map/volume.py", line 1664, in displayed_matrices

matrices.append(self.matrix(read_matrix))

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/map/volume.py", line 1179, in matrix

m = self.region_matrix(r, read_matrix)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/map/volume.py", line 1203, in region_matrix

m = d.matrix(origin, size, step, progress, from_cache_only)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/map/data/griddata.py", line 249, in matrix

m = self._read_full_planes(ijk_origin, ijk_size, ijk_step, progress)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/map/data/griddata.py", line 274, in _read_full_planes

fm = self.read_matrix(f_ijk_origin, f_ijk_size, f_ijk_step, progress)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/dicom/dicom_grid.py", line 119, in read_matrix

self.time, c, m, progress)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/dicom/dicom_format.py", line 539, in read_matrix

p = self.read_plane(k, time, channel)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/dicom/dicom_format.py", line 558, in read_plane

data = d.pixel_array

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pydicom/dataset.py", line 929, in pixel_array

self.convert_pixel_data()

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pydicom/dataset.py", line 875, in convert_pixel_data

raise last_exception

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pydicom/dataset.py", line 843, in convert_pixel_data

arr = handler.get_pixeldata(self)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pydicom/pixel_data_handlers/pillow_handler.py", line 197, in get_pixeldata

decompressed_image = Image.open(fio)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/PIL/Image.py", line 2675, in open

im = _open_core(fp, filename, prefix)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/PIL/Image.py", line 2659, in _open_core

im = factory(fp, filename)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/PIL/JpegImagePlugin.py", line 779, in jpeg_factory

im = JpegImageFile(fp, filename)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/PIL/ImageFile.py", line 103, in init

self._open()

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/PIL/JpegImagePlugin.py", line 338, in _open

handler(self, i)

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/PIL/JpegImagePlugin.py", line 166, in SOF

raise SyntaxError("cannot handle %d-bit layers" % self.bits)

File "<string>", line None

SyntaxError: cannot handle 16-bit layers

Error processing trigger "graphics update": cannot handle 16-bit layers:
File "", line None
SyntaxError: cannot handle 16-bit layers

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/PIL/JpegImagePlugin.py", line 166, in SOF
raise SyntaxError("cannot handle %d-bit layers" % self.bits)

See log for complete Python traceback.

comment:4 by Tom Goddard, 7 years ago

PyDICOM documentation claims it handles lossless jpeg format (1.2.840.10008.1.2.4.70)

​https://pydicom.github.io/pydicom/dev/image_data_handlers.html

but it appears that is incorrect at least for 16-bit images. Possibly it does handle it for 8-bit images. But most DICOM grayscale data is 16-bit.

comment:5 by Tom Goddard, 7 years ago

Apparently the main option for pydicom to read lossless jpeg is to install another library gdcm.

comment:6 by Tom Goddard, 7 years ago

Reported this problem at pydicom github

​https://github.com/pydicom/pydicom/issues/813

The limitation is really in Pillow and I did not find any github pillow issues about reading 16-bit jpeg and it is clear from the PIL/JpegImagePlugin.py code that it only handles 8-bit images. So a fix would have to be in Pillow, or other means of reading the 16-bit jpeg such as using the GDCM library would be needed.

comment:7 by Tom Goddard, 6 years ago

pydicom corrected their documentation to note that it cannot read 16-bit lossless jpeg using PIL.

Made the ChimeraX DICOM reader issue a warning on such files and not read them, instead of giving a traceback. Will need to use the GDCM library to read these DICOM files.

comment:8 by Tom Goddard, 6 years ago

Type: defect → enhancement

comment:9 by Tom Goddard, 5 years ago

I added gdcm to ChimeraX on November 28, 2019 and forgot to note it in this ticket. Tests show it was working and allowed ChimeraX to open the attached DICOM files in ChimeraX 0.92 from March 2020, but it fails to open in ChimeraX 1.0 and 1.1 because gdcm will not import in those later ChimeraX versions. The reason gdcm import fails is because of missing libopenjp2.7.dylib on Mac. If I add that library to gdcm-2.8.9/lib in ChimeraX 1.1 then gdcm works and the attached dicom can be loaded. Using Mac dtruss to trace system calls when running ChimeraX 0.92 shows it finds /usr/local/lib/libopenjp2.7 but for some reason the newer ChimeraX does not search /usr/local/lib. At any rate that library is from homebrew and would allow using ChimeraX 0.92 on another Mac to use gdcm.

I am trying to update gdcm to Python 3.8 and that is when I found the above problem.

comment:10 by Tom Goddard, 5 years ago

I got the github gdcm 3.0.8 release to work with Python 3.8 Mac ChimeraX after a bunch of library path debugging. I reported the library path problems in this gdcm bug report

​https://sourceforge.net/p/gdcm/bugs/510/

comment:11 by Elaine Meng, 3 years ago

Component: Volume Data → DICOM

comment:12 by Tom Goddard, 3 years ago

Owner: changed from Tom Goddard to Zach Pearson

Ticket #3758 is about getting gdcm to work with Python 3.9.

by Zach Pearson, 3 years ago

Attachment: Screenshot 2022-06-22 at 17.54.03.png​ added

I installed python-gdcm==3.0.12 (supports up to Python 3.10) into my ChimeraX on my laptop (arm64 macOS Ventura beta 2) and it opens the attached file with no complaints.

comment:13 by Zach Pearson, 3 years ago

Resolution: → fixed
Status: assigned → closed

(Same resolution as #3758) Added python-gdcm==3.0.12 to app_requirements.txt

in reply to:  16 comment:14 by goddard@…, 3 years ago 

I think we should consider if this is the right way to distribute this dicom capability.  The python-gdcm package for Python 3.9 on windows is 26 Mbytes compressed, increasing the size of our Windows distribution (240 Mbytes) by 10% to read a file format that not a single bug report requested in some years.  On Mac and Linux the increase is less, about 5%.   I think we need to consider how to avoid bloating the software for such extremely rarely used features.  I'd be in favor of not distributing this right now.  It is great that you found a solution.  But maybe we should use this as a trial case of how to distribute rarely needed capabilities via Toolshed.

Another idea for supporting this DICOM format without making every user download a heftier ChimeraX would be to add to the error message that detects the lossless 16-bit JPEG, instructing the user to type command "pip install python-gdcm" to the ChimeraX command line to get support for that DICOM format.  Of course that command does not exist in ChimeraX.  But this would be a motivation to add the pip install capability to ChimeraX (the subject of another ticket).

comment:15 by Zach Pearson, 3 years ago

Let's talk about it in the next developer meeting a little bit. We've got a pretty strong plugin architecture and have all the scaffolding available to move to an a-la-carte distribution scheme for bundles, so we should discuss what we're going to do with it. I'm picturing something like Visual Studio's installer, which lets developers pick only what they need from a variety of tools.

in reply to:  18 ; comment:16 by goddard@…, 3 years ago 

I think our Toolshed is lacking as today's discussion of putting the Basic Actions tool fix revealed (ticket #7123).  While we can discuss at our next developer meeting, I think you could move this along nicely by actually putting the dicom lossless jpeg support on Toolshed and making the DICOM reader error message tell the user how to get it.  Can a tool that has no code of its own but simply lists python-gdcm as a dependency get it installed?  Seems possible, but I wouldn't bet on it until someone tries it.  I also think providing the user a simple ChimeraX command (not shell command) to install python-gdcm would be an interesting approach to try.

comment:17 by pett, 3 years ago

A more generic solution would be to use an import hook. As a reminder an import hook is called only if the normal import machinery has failed to locate a module and would otherwise throw an ImportError. The import hook can look at what module was being requested, determine if that module is provided on the Toolshed, and if it is inform the user that they need to install a particular bundle to carry out their requested operation, bring up the bundle's Toolshed page in the help browser, and raise CancelOperation to stop the current code execution. If the module isn't provided by the Toolshed, just carry on and raise ImportError.

Last edited 3 years ago by pett (previous) (diff)

in reply to:  20 ; comment:18 by goddard@…, 3 years ago 

I am not in favor of adding an import hook.  I prefer not to tamper with every Python import in order to handle a rare situation.  That can raise future debugging problems.

comment:19 by pett, 3 years ago

A close reading of what I wrote reveals that the import hook is only executed if an ImportError is about to be raised, not on every import.

Last edited 3 years ago by pett (previous) (diff)

in reply to:  22 ; comment:20 by goddard@…, 3 years ago 

I got it.  I think an import hook that engages on every import error throughout ChimeraX is ill-advised, if our use case so far is handling 16-bit lossless jpeg dicoms.  I am a fan of having localized solutions to narrow problems.  But if we get to the point where we have 20 optional toolshed packages that are imported in various places, it would be sensible to consider the import hook mechanism.  But I prefer not to hide these tricky mechanisms.  If code wants to use an optional package that may not be installed it would be confusing for me to understand how it works if I saw "import losslessdicom".  I'd probably prefer to see something more transparent like "if optional_package_available('losslessdicom'): proceed to use it".

comment:21 by pett, 3 years ago

Having a generic mechanism like this could allow us to offload more rarely used capabilities to the Toolshed (perhaps even the entirety of DICOM).

I certainly prefer it over a mechanism where the coder has to know whether or not a package is shipped pre-installed, and has to learn a custom API if it isn't.

comment:22 by Zach Pearson, 3 years ago

if optional_package_available is basically how other packages handle optional dependencies. We'd be taking that one step further offering to install it for the user.

_has_dep = False
try:
    import optional_dep
    _has_dep = True
except ImportError:
    _has_dep = False

...

# Code that warns the user if they hit a codepath that uses the dependency.

This could even be a decorator. 

def uses_optional(dep, func):
    try: 
        import dep
        return func
    except ImportError:
        def warn_and_attempt_install_instead():
            ... # Warn the user they need to install something, or prompt them with a popup and with consent try to install it from the toolshed
        return warn_and_attempt_install_instead

in reply to:  25 comment:23 by goddard@…, 3 years ago 

Right, the idea is optional_package_available() would raise dialog asking the user if they want to install it if not yet installed, hopefully with a single button press.

I think it is silly to spend much time talking about the mechanism (which we have discussed various times starting years ago).  What we need is to try a simple case where it makes sense like lossless dicom support and see how it flops.  Will Toolshed be able to handle this simple situation where all that is needed is a PyPi install of python-gdcm, and handle the errors when it is not available for the user's platform?  The code is used to present the button that installs that is a triviality compared to wrangling Toolshed.

comment:24 by pett, 3 years ago

Cc: pett added

comment:25 by pett, 3 years ago

It's certainly true that either approach will probably require substantial enhancements to Toolshed capabilities.

comment:26 by pett, 3 years ago

Cc: Greg Couch added

Should this ticket still be in "closed" status?

comment:27 by Zach Pearson, 3 years ago

Yes. I will open a new ticket about installing optional dependencies from Toolshed and/or pip.

in reply to:  30 comment:28 by goddard@…, 3 years ago 

My comment many comments ago that increasing the ChimeraX Windows distribution size by 10% (25 Mbytes compressed added to 240 Mbytes) by adding gdcm to handle a lossless jpeg dicom that no-one other than Meghan some years ago has used was meant to suggest that gdcm should be taken out of the build because it does considerably more harm than good.  The ticket could remain open (or closed and replaced by a new ticket) to in the future possibly provide a solution via toolshed.

comment:29 by Zach Pearson, 3 years ago

I don't want to have to keep a mapping in my head from a ticket's title issue to its actual issue. We know the solution is to use the new version of gdcm, so let's open another one about getting dependencies on the fly from the toolshed and reducing the apparent size of the ChimeraX distribution.

comment:30 by pett, 3 years ago

A new ticket sounds like the correct choice.

in reply to:  33 comment:31 by goddard@…, 3 years ago 

I too always favor new tickets.

But let's not close and forget this one.  Either take gdcm out of the builds, or make a statement that we are going to accept big increases in distribution size for features nobody is using.  As I've said, my preference is to leave gdcm out.  It is great that gdcm is now available for Python 3.9 -- an important discovery.  But that does not mean it makes sense to use it.

comment:32 by Zach Pearson, 3 years ago

If we have a fix we should leave it in until a workable alternative is implemented. If a feature is so rarely used that we've gotten no bug reports about it in years, it's possibly a signal it should be deprecated. Is anyone really hard up for 25Mb of disk space?

in reply to:  35 ; comment:33 by goddard@…, 3 years ago 

The trouble is 25 Mbytes of network bandwidth and wasted time of everyone downloading stuff that not even 1 in 10000 users will benefit from.  The goal is to deliver useful software, not close a 3 year old ticket (that I created) at any cost.

comment:34 by Zach Pearson, 3 years ago

Resolution: fixed
Status: closed → reopened

comment:35 by Zach Pearson, 3 years ago

Owner: changed from Zach Pearson to Tom Goddard
Status: reopened → assigned

It sounds like you've made up your mind, and I disagree with you, but do what you think is right.

comment:36 by Tom Goddard, 3 years ago

Owner: changed from Tom Goddard to Zach Pearson

I won't revert your changes. I've added it to the Monday developer meeting agenda.

comment:37 by Greg Couch, 3 years ago

I'm for leaving python-gdcm out ChimeraX. I'd leave DICOM out of the regular distribution and just have it on the Toolshed too. python-gdcm could be a non-optional dependency for the Toolshed bundle.

comment:38 by Zach Pearson, 3 years ago

Resolution: → wontfix
Status: assigned → closed

After the developers meeting, the agreed upon resolution was not to solve it at the DICOM bundle level. It's such a rarely used feature that it doesn't justify bumping the release size by 25MB. Instead, we'll warn the user when they try to open a 16-bit lossless JPEG that they could do it, *if* they installed python-gdcm, and give them instructions for how (or perhaps provide a command to do so, see #4762).

Note: See TracTickets for help on using tickets.