#10365 closed defect (fixed)

Warnings at startup: Corrupt JPEG data: 3 extraneous bytes before marker 0xd9

Reported by: goddard@… Owned by: Tom Goddard
Priority: normal Milestone:
Component: Input/Output Version:
Keywords: Cc: pett
Blocked By: Blocking:
Notify when closed: Platform: all
Project: ChimeraX

Description

The following bug report has been submitted:
Platform:        macOS-14.1.2-arm64-arm-64bit
ChimeraX Version: 1.8.dev202312190520 (2023-12-19 05:20:12 UTC)
Description
Starting ChimeraX from a shell on this Mac always produces these 3 warnings in the shell

Corrupt JPEG data: 3 extraneous bytes before marker 0xd9
Corrupt JPEG data: 5 extraneous bytes before marker 0xd9
Corrupt JPEG data: 1 extraneous bytes before marker 0xd9

These appear to be generated by the thumbnail images in the recent files history.  The "marker 0xd9" is the end of image marker of a jpeg image, so the message indicates there are some unexpected bytes before the end of image.  I never saw these until recently so it is likely that a recent ChimeraX version created them.

Log:
UCSF ChimeraX version: 1.8.dev202312190520 (2023-12-19)  
© 2016-2023 Regents of the University of California. All rights reserved.  
How to cite UCSF ChimeraX  

> help help:contact.html




OpenGL version: 4.1 Metal - 88
OpenGL renderer: Apple M1 Max
OpenGL vendor: Apple

Python: 3.11.2
Locale: en_US.UTF-8
Qt version: PyQt6 6.3.1, Qt 6.3.1
Qt runtime version: 6.3.2
Qt platform: cocoa
Hardware:

    Hardware Overview:

      Model Name: MacBook Pro
      Model Identifier: MacBookPro18,2
      Model Number: MK1H3LL/A
      Chip: Apple M1 Max
      Total Number of Cores: 10 (8 performance and 2 efficiency)
      Memory: 32 GB
      System Firmware Version: 10151.41.12
      OS Loader Version: 10151.41.12

Software:

    System Software Overview:

      System Version: macOS 14.1.2 (23B92)
      Kernel Version: Darwin 23.1.0
      Time since boot: 17 days, 7 hours, 52 minutes

Graphics/Displays:

    Apple M1 Max:

      Chipset Model: Apple M1 Max
      Type: GPU
      Bus: Built-In
      Total Number of Cores: 32
      Vendor: Apple (0x106b)
      Metal Support: Metal 3
      Displays:
        Color LCD:
          Display Type: Built-in Liquid Retina XDR Display
          Resolution: 3456 x 2234 Retina
          Main Display: Yes
          Mirror: Off
          Online: Yes
          Automatically Adjust Brightness: No
          Connection Type: Internal


Installed Packages:
    alabaster: 0.7.13
    appdirs: 1.4.4
    appnope: 0.1.3
    asttokens: 2.4.1
    Babel: 2.13.1
    backcall: 0.2.0
    beautifulsoup4: 4.11.2
    blockdiag: 3.0.0
    blosc2: 2.0.0
    build: 0.10.0
    certifi: 2022.12.7
    cftime: 1.6.3
    charset-normalizer: 3.3.2
    ChimeraX-AddCharge: 1.5.13
    ChimeraX-AddH: 2.2.5
    ChimeraX-AlignmentAlgorithms: 2.0.1
    ChimeraX-AlignmentHdrs: 3.4.2
    ChimeraX-AlignmentMatrices: 2.1
    ChimeraX-Alignments: 2.12.2
    ChimeraX-AlphaFold: 1.0
    ChimeraX-AltlocExplorer: 1.1.1
    ChimeraX-AmberInfo: 1.0
    ChimeraX-Arrays: 1.1
    ChimeraX-Atomic: 1.51
    ChimeraX-AtomicLibrary: 12.1.3
    ChimeraX-AtomSearch: 2.0.1
    ChimeraX-AxesPlanes: 2.4
    ChimeraX-BasicActions: 1.1.2
    ChimeraX-BILD: 1.0
    ChimeraX-BlastProtein: 2.1.2
    ChimeraX-BondRot: 2.0.4
    ChimeraX-BugReporter: 1.0.1
    ChimeraX-BuildStructure: 2.10.5
    ChimeraX-Bumps: 1.0
    ChimeraX-BundleBuilder: 1.2.2
    ChimeraX-ButtonPanel: 1.0.1
    ChimeraX-CageBuilder: 1.0.1
    ChimeraX-CellPack: 1.0
    ChimeraX-Centroids: 1.4
    ChimeraX-ChangeChains: 1.1
    ChimeraX-CheckWaters: 1.3.2
    ChimeraX-ChemGroup: 2.0.1
    ChimeraX-Clashes: 2.2.4
    ChimeraX-ColorActions: 1.0.3
    ChimeraX-ColorGlobe: 1.0
    ChimeraX-ColorKey: 1.5.5
    ChimeraX-CommandLine: 1.2.5
    ChimeraX-ConnectStructure: 2.0.1
    ChimeraX-Contacts: 1.0.1
    ChimeraX-Core: 1.8.dev202312190520
    ChimeraX-CoreFormats: 1.2
    ChimeraX-coulombic: 1.4.2
    ChimeraX-Crosslinks: 1.0
    ChimeraX-Crystal: 1.0
    ChimeraX-CrystalContacts: 1.0.1
    ChimeraX-DataFormats: 1.2.3
    ChimeraX-Dicom: 1.2
    ChimeraX-DistMonitor: 1.4
    ChimeraX-DockPrep: 1.1.3
    ChimeraX-Dssp: 2.0
    ChimeraX-EMDB-SFF: 1.0
    ChimeraX-ESMFold: 1.0
    ChimeraX-FileHistory: 1.0.1
    ChimeraX-FunctionKey: 1.0.1
    ChimeraX-Geometry: 1.3
    ChimeraX-gltf: 1.0
    ChimeraX-Graphics: 1.1.1
    ChimeraX-Hbonds: 2.4
    ChimeraX-Help: 1.2.2
    ChimeraX-HKCage: 1.3
    ChimeraX-IHM: 1.1
    ChimeraX-ImageFormats: 1.2
    ChimeraX-IMOD: 1.0
    ChimeraX-IO: 1.0.1
    ChimeraX-ItemsInspection: 1.0.1
    ChimeraX-IUPAC: 1.0
    ChimeraX-Label: 1.1.8
    ChimeraX-ListInfo: 1.2.2
    ChimeraX-Log: 1.1.6
    ChimeraX-LookingGlass: 1.1
    ChimeraX-Maestro: 1.9.1
    ChimeraX-Map: 1.1.4
    ChimeraX-MapData: 2.0
    ChimeraX-MapEraser: 1.0.1
    ChimeraX-MapFilter: 2.0.1
    ChimeraX-MapFit: 2.0
    ChimeraX-MapSeries: 2.1.1
    ChimeraX-Markers: 1.0.1
    ChimeraX-Mask: 1.0.2
    ChimeraX-MatchMaker: 2.1.2
    ChimeraX-MCopy: 1.0
    ChimeraX-MDcrds: 2.6
    ChimeraX-MedicalToolbar: 1.0.2
    ChimeraX-Meeting: 1.0.1
    ChimeraX-MLP: 1.1.1
    ChimeraX-mmCIF: 2.12.1
    ChimeraX-MMTF: 2.2
    ChimeraX-Modeller: 1.5.13
    ChimeraX-ModelPanel: 1.4
    ChimeraX-ModelSeries: 1.0.1
    ChimeraX-Mol2: 2.0.3
    ChimeraX-Mole: 1.0
    ChimeraX-Morph: 1.0.2
    ChimeraX-MouseModes: 1.2
    ChimeraX-Movie: 1.0
    ChimeraX-Neuron: 1.0
    ChimeraX-Nifti: 1.1
    ChimeraX-NRRD: 1.1
    ChimeraX-Nucleotides: 2.0.3
    ChimeraX-OpenCommand: 1.13.1
    ChimeraX-PDB: 2.7.3
    ChimeraX-PDBBio: 1.0.1
    ChimeraX-PDBLibrary: 1.0.3
    ChimeraX-PDBMatrices: 1.0
    ChimeraX-PickBlobs: 1.0.1
    ChimeraX-Positions: 1.0
    ChimeraX-PresetMgr: 1.1
    ChimeraX-PubChem: 2.1
    ChimeraX-ReadPbonds: 1.0.1
    ChimeraX-Registration: 1.1.2
    ChimeraX-RemoteControl: 1.0
    ChimeraX-RenderByAttr: 1.2
    ChimeraX-RenumberResidues: 1.1
    ChimeraX-ResidueFit: 1.0.1
    ChimeraX-RestServer: 1.2
    ChimeraX-RNALayout: 1.0
    ChimeraX-RotamerLibMgr: 4.0
    ChimeraX-RotamerLibsDunbrack: 2.0
    ChimeraX-RotamerLibsDynameomics: 2.0
    ChimeraX-RotamerLibsRichardson: 2.0
    ChimeraX-SaveCommand: 1.5.1
    ChimeraX-SchemeMgr: 1.0
    ChimeraX-SDF: 2.0.2
    ChimeraX-Segger: 1.0
    ChimeraX-Segment: 1.0.1
    ChimeraX-SelInspector: 1.0
    ChimeraX-SeqView: 2.11
    ChimeraX-Shape: 1.0.1
    ChimeraX-Shell: 1.0.1
    ChimeraX-Shortcuts: 1.1.1
    ChimeraX-ShowSequences: 1.0.2
    ChimeraX-SideView: 1.0.1
    ChimeraX-Smiles: 2.1.2
    ChimeraX-SmoothLines: 1.0
    ChimeraX-SpaceNavigator: 1.0
    ChimeraX-StdCommands: 1.14.1
    ChimeraX-STL: 1.0.1
    ChimeraX-Storm: 1.0
    ChimeraX-StructMeasure: 1.1.2
    ChimeraX-Struts: 1.0.1
    ChimeraX-Surface: 1.0.1
    ChimeraX-SwapAA: 2.0.1
    ChimeraX-SwapRes: 2.2.2
    ChimeraX-TapeMeasure: 1.0
    ChimeraX-TaskManager: 1.0
    ChimeraX-Test: 1.0
    ChimeraX-Toolbar: 1.1.2
    ChimeraX-ToolshedUtils: 1.2.4
    ChimeraX-Topography: 1.0
    ChimeraX-ToQuest: 1.0
    ChimeraX-Tug: 1.0.1
    ChimeraX-UI: 1.33.4
    ChimeraX-uniprot: 2.3
    ChimeraX-UnitCell: 1.0.1
    ChimeraX-ViewDockX: 1.3.2
    ChimeraX-VIPERdb: 1.0
    ChimeraX-Vive: 1.1
    ChimeraX-VolumeMenu: 1.0.1
    ChimeraX-vrml: 1.0
    ChimeraX-VTK: 1.0
    ChimeraX-WavefrontOBJ: 1.0
    ChimeraX-WebCam: 1.0.2
    ChimeraX-WebServices: 1.1.3
    ChimeraX-Zone: 1.0.1
    colorama: 0.4.6
    comm: 0.2.0
    contourpy: 1.2.0
    cxservices: 1.2.2
    cycler: 0.12.1
    Cython: 0.29.33
    debugpy: 1.8.0
    decorator: 5.1.1
    docutils: 0.19
    executing: 2.0.1
    filelock: 3.9.0
    fonttools: 4.45.1
    funcparserlib: 2.0.0a0
    glfw: 2.6.3
    grako: 3.16.5
    h5py: 3.10.0
    html2text: 2020.1.16
    idna: 3.6
    ihm: 0.38
    imagecodecs: 2023.9.18
    imagesize: 1.4.1
    ipykernel: 6.23.2
    ipython: 8.14.0
    ipython-genutils: 0.2.0
    ipywidgets: 8.1.1
    jedi: 0.18.2
    Jinja2: 3.1.2
    jupyter-client: 8.2.0
    jupyter-core: 5.5.0
    jupyterlab-widgets: 3.0.9
    kiwisolver: 1.4.5
    line-profiler: 4.0.2
    lxml: 4.9.2
    lz4: 4.3.2
    MarkupSafe: 2.1.3
    matplotlib: 3.7.2
    matplotlib-inline: 0.1.6
    msgpack: 1.0.4
    nest-asyncio: 1.5.8
    netCDF4: 1.6.2
    networkx: 3.1
    nibabel: 5.0.1
    nptyping: 2.5.0
    numexpr: 2.8.7
    numpy: 1.25.1
    openvr: 1.23.701
    packaging: 21.3
    ParmEd: 3.4.3
    parso: 0.8.3
    pep517: 0.13.0
    pexpect: 4.9.0
    pickleshare: 0.7.5
    Pillow: 10.0.1
    pip: 23.0
    pkginfo: 1.9.6
    platformdirs: 4.0.0
    prompt-toolkit: 3.0.41
    psutil: 5.9.5
    ptyprocess: 0.7.0
    pure-eval: 0.2.2
    py-cpuinfo: 9.0.0
    pycollada: 0.7.2
    pydicom: 2.3.0
    Pygments: 2.16.1
    pynrrd: 1.0.0
    PyOpenGL: 3.1.7
    PyOpenGL-accelerate: 3.1.7
    pyopenxr: 1.0.2801
    pyparsing: 3.0.9
    pyproject-hooks: 1.0.0
    PyQt6: 6.3.1
    PyQt6-Qt6: 6.3.2
    PyQt6-sip: 13.4.0
    PyQt6-WebEngine: 6.3.1
    PyQt6-WebEngine-Qt6: 6.3.2
    python-dateutil: 2.8.2
    pytz: 2023.3.post1
    pyzmq: 25.1.1
    qtconsole: 5.4.3
    QtPy: 2.4.1
    RandomWords: 0.4.0
    requests: 2.31.0
    scipy: 1.11.1
    setuptools: 67.4.0
    setuptools-scm: 7.0.5
    sfftk-rw: 0.7.3
    six: 1.16.0
    snowballstemmer: 2.2.0
    sortedcontainers: 2.4.0
    soupsieve: 2.5
    sphinx: 6.1.3
    sphinx-autodoc-typehints: 1.22
    sphinxcontrib-applehelp: 1.0.7
    sphinxcontrib-blockdiag: 3.0.0
    sphinxcontrib-devhelp: 1.0.5
    sphinxcontrib-htmlhelp: 2.0.4
    sphinxcontrib-jsmath: 1.0.1
    sphinxcontrib-qthelp: 1.0.6
    sphinxcontrib-serializinghtml: 1.1.9
    stack-data: 0.6.3
    superqt: 0.5.0
    tables: 3.8.0
    tcia-utils: 1.5.1
    tifffile: 2023.7.18
    tinyarray: 1.2.4
    tomli: 2.0.1
    tornado: 6.4
    traitlets: 5.9.0
    typing-extensions: 4.8.0
    tzdata: 2023.3
    urllib3: 2.1.0
    wcwidth: 0.2.12
    webcolors: 1.12
    wheel: 0.38.4
    wheel-filename: 1.4.1
    widgetsnbextension: 4.0.9

Attachments (7)

169.jpg (4.8 KB ) - added by Tom Goddard 22 months ago.
bad jpeg
170.jpg (2.6 KB ) - added by Tom Goddard 22 months ago.
bad jpeg
177.jpg (5.4 KB ) - added by Tom Goddard 22 months ago.
bad jpeg
171.jpg (4.7 KB ) - added by Tom Goddard 22 months ago.
good jpeg
172.jpg (1.9 KB ) - added by Tom Goddard 22 months ago.
good jpeg
173.jpg (2.8 KB ) - added by Tom Goddard 22 months ago.
good jpeg
jpg_test.zip (3.0 MB ) - added by Tom Goddard 22 months ago.
Test png files and gen.py script that creates jpg copies that are corrupted.

Change History (25)

comment:1 by Tom Goddard, 22 months ago

Cc: pett added
Component: UnassignedInput/Output
Owner: set to Tom Goddard
Platform: all
Project: ChimeraX
Status: newassigned
Summary: ChimeraX bug report submissionWarnings at startup: Corrupt JPEG data: 3 extraneous bytes before marker 0xd9

The ChimeraX file history thumbnail image save as JPEG is handled by the Python Image Library (pillow) Image.save(path, format = 'JPEG') method. ChimeraX 1.6 used Pillow version 9.3.0 and ChimeraX 1.7 uses Pillow version 10.0.1. My guess is the problem JPEG thumbnails were introduced by Pillow 10.0.1.

comment:2 by Tom Goddard, 22 months ago

I exported the 179 jpeg images from my file history as individual files and used ImageMagick to identify the corrupt ones

$ identify -verbose *.jpg > id
identify: Corrupt JPEG data: 1 extraneous bytes before marker 0xd9 `169.jpg' @ warning/jpeg.c/JPEGWarningHandler/403.
identify: Corrupt JPEG data: 3 extraneous bytes before marker 0xd9 `170.jpg' @ warning/jpeg.c/JPEGWarningHandler/403.
identify: Corrupt JPEG data: 5 extraneous bytes before marker 0xd9 `177.jpg' @ warning/jpeg.c/JPEGWarningHandler/403.

New history files are added at the end so these are among the 11 most recent files, supporting my suspicion that this is probably a relatively recently introduced bug.

comment:3 by Tom Goddard, 22 months ago

Dumping the markers from one of the bad files 177.jpg shows that the block with the extra bytes is an SOS marker with entropy-coded data. This blocks are the same for the other good and bad thumbnails, always the last one is entropy-coded data. The size of every thumbnail is 128 x 128.

$ python3 ~/Downloads/jpegdump.py 177.jpg 
File: 177.jpg
  1 p=0x00000000    : m=ffd8 SOI
  2 p=0x00000002 d=0: m=ffe0 APP0  l=   16 e=1.985228 a=6.615385
  3 p=0x00000014 d=0: m=ffdb DQT   l=   67 e=5.095219 a=4.718750 remark: 65/65 = 1.000000
  4 p=0x00000059 d=0: m=ffdb DQT   l=   67 e=1.375467 a=1.828125 remark: 65/65 = 1.000000
  5 p=0x0000009e d=0: m=ffc0 SOF0  l=   17 e=2.872906 a=46.500000 remark: p=8 h=128 w=128 c=3
  6 p=0x000000b1 d=0: m=ffc4 DHT   l=   31 e=2.815937 a=0.750000
  7 p=0x000000d2 d=0: m=ffc4 DHT   l=  181 e=7.270047 a=13.168539
  8 p=0x00000189 d=0: m=ffc4 DHT   l=   31 e=2.799180 a=0.642857
  9 p=0x000001aa d=0: m=ffc4 DHT   l=  181 e=7.243483 a=18.140449
 10 p=0x00000261 d=0: m=ffda SOS   l=   12 e=2.446439 a=21.222222 remark: c=3
                                  entropy-coded data: l=4856 e=7.908492 a=82.117817 #ff00=16
 11 p=0x00001567 d=0: m=ffd9 EOI

Dumping the 177.jpg in hex shows the last bytes are

$ od -t x1 177.jpg | tail
0012340    77  73  d0  9e  7d  b3  d2  a2  2f  fb  95  8c  67  a9  63  90
0012360    3a  fb  52  02  ca  84  06  21  5b  a8  07  ae  3d  45  4c  96
0012400    97  77  36  f7  17  91  da  ca  f6  f6  fb  7c  f9  63  88  ec
0012420    8b  71  c2  ee  20  61  72  78  19  eb  55  3a  ae  de  f3  fe
0012440    98  ec  57  a9  e5  bc  9e  6b  5b  7b  57  93  f7  16  fb  bc
0012460    b8  c0  00  02  c7  2c  c7  1d  58  e0  0c  9e  70  aa  33  85
0012500    00  47  14  32  dc  39  48  62  79  18  2b  39  54  52  48  55
0012520    05  98  f1  d8  00  49  3d  80  26  99  59  b4  9b  d7  a0  00
0012540    00  00  00  00  00  00  07  ff  d9                            
0012551

The 3 bad files all have seven or eight 0 bytes, then a non-zero, then ff d9. Several good files I looked add never had a string of 0 bytes at the end.

by Tom Goddard, 22 months ago

Attachment: 169.jpg added

bad jpeg

by Tom Goddard, 22 months ago

Attachment: 170.jpg added

bad jpeg

by Tom Goddard, 22 months ago

Attachment: 177.jpg added

bad jpeg

by Tom Goddard, 22 months ago

Attachment: 171.jpg added

good jpeg

by Tom Goddard, 22 months ago

Attachment: 172.jpg added

good jpeg

by Tom Goddard, 22 months ago

Attachment: 173.jpg added

good jpeg

comment:4 by Tom Goddard, 22 months ago

To save the jpeg files in json the code base64 encodes the bytes the converts to utf-8 to produce a string.

image_base64 = codecs.encode(image_bytes, 'base64').decode('utf-8')

The base64 bytes are ascii code for A-Z, a-z, 0-9, /, + and also new line characters to divide into lines of less than or equal to 76 bytes (mime base64 standard) and those newlines are ignored. I don't think the UTF-8 encoding and decoding will corrupt such a byte string.

comment:5 by Tom Goddard, 22 months ago

The PIL version on my development build that very likely was what made image 177.jpg is 10.0.1. I updated PIL from 10.0.0 to 10.0.1 on October 10, 2023.

I tried generating a bad thumbnail image using Python with PIL 10.0.1 by converting my 179 thumbnails to PNG then reading those in PIL and saving them as JPEG. But none of the saved JPEG images were corrupted. I also tried base64 encoding then convert to utf-8 and reversing on the jpg byte stream, and still all images are fine.

ChimeraX creates the PIL images from a numpy array, and maybe that is necessary for PIL to save a bad JPEG.

pi = Image.fromarray(rgba[::-1, :, :ncomp])

I read all my PNG images into numpy arrays, flipped vertically and saved as jpeg files and none were corrupted.

All these attempts to make a corrupt jpeg were in a locally installed Python 3.10.12, not the ChimeraX Python and PIL.

Aha! I ran the same test converting PNG to numpy array and saving as JPEG within my ChimeraX development build and 66 of them came out corrupted 1/4 with extraneous bytes before 0xd9 and 3/4 with "premature end of data segment". Same problem with the Dec 15, 2023 ChimeraX 1.7 release candidate. ChimeraX 1.6.1 does not produce any bad JPEG files.

comment:6 by Tom Goddard, 22 months ago

My local Python 3.10.12 is using numpy 1.26.2 while by ChimeraX devel build is using numpy 1.25.1. Possibly that is the key difference. Updating ChimeraX numpy to 1.26.2 did not fix the problem. The local python I am using is from conda and the conda lib directory has libjpeg.8.2.2.dylib. ChimeraX does not have any libjpeg in its lib directory. ChimeraX does have a PIL/.dylibs/libjpeg.62.4.0.dylib. The local conda Python does not have PIL/.dylibs. Sym linking ChimeraX to use the conda libjpeg fails because of a missing symbol. Pillow 10.1.0 is out but uses same libjpeg and installing it in ChimeraX give the same corrupt jpeg files. Installing Pillow 9.5.0 in ChimeraX fixes the corrupt jpegs. That version does not have PIL/.dylibs/libjpeg*.dylib and instead has PIL/.dylibs/libopenjp2.2.5.0.dylib.

So it looks like Pillow 10 has a bad libjpeg. Tested in local homebrew python 3.11 with Pillow 10.0.1 and indeed it writes the same corrupted jpeg files.

comment:7 by Tom Goddard, 22 months ago

Maybe we should fix Pillow before the ChimeraX 1.7 release. Although I have not seen it cause problems with the thumbnails, if users save jpg images for presentations those might cause problems when imported into other software.

comment:8 by Tom Goddard, 22 months ago

ChimeraX 1.6 used Pillow 9.3.0 which would probably be safe. But since image saving is critical we would want some testing of that.

comment:9 by pett, 22 months ago

Well, 1.7 is out. We could make a 1.7.1 possibly, maybe in January?

comment:10 by Tom Goddard, 22 months ago

Probably won't need to fix this until ChimeraX 1.8.

The file history thumbnails where we see this problem don't seem to be a big enough problem to worry about. I'm a little concerned that users may be saving jpeg images for presentations that could be effected. But since no one has reported it so far that probably also isn't a problem.

comment:11 by Tom Goddard, 22 months ago

I'm wondering if the same bug afflicts Intel Mac and Windows and Linux. I'll attach my test case of 179 PNG images (made by converting my file history jpg to png with ImageMagick mogrify) and Python script gen.py which uses PIL to copy those PNG images to JPEG images. Running ImageMagick "identify -verbose *.jpg > out" on those outputs 65 warning messages to the shell like

identify: Corrupt JPEG data: 2 extraneous bytes before marker 0xd9 `165.jpg' @ warning/jpeg.c/JPEGWarningHandler/403.
identify: Corrupt JPEG data: premature end of data segment `166.jpg' @ warning/jpeg.c/JPEGWarningHandler/403.

when using PIL 10.0.1 on Mac ARM, macOS 14.1.2.

by Tom Goddard, 22 months ago

Attachment: jpg_test.zip added

Test png files and gen.py script that creates jpg copies that are corrupted.

comment:12 by Tom Goddard, 22 months ago

This bug was introduced in ChimeraX on October 10, 2023.

I updated Pillow from version 9.3.0 to 10.0.0 on July 17, 2023, and then updated again to version 10.0.1 on October 10, 2023 (to suppress a GitHub security warning about 10.0.1). Version 10.0.0 does not produce the corrupted jpg images. PIL 10.0.0 includes libjpeg.62.3.0.dylib while 10.0.1 includes libjpeg.62.4.0.dylib. PIL 10.1.0 (released Oct 15, 2023) is the current PyPi Pillow and also uses libjpeg.62.4.0.dylib and produces corrupt JPEG images.

comment:13 by Tom Goddard, 22 months ago

This probably the following Pillow bug

https://github.com/python-pillow/Pillow/issues/7625

It blames the problem on a library libjpeg-turbo which I do not see in the Mac ARM Pillow (unless it is part of libopenjp*.dylib, but that has the same version libopenjp2.2.5.0.dylib in both the broken 10.0.1 and working 10.0.0). So I'm not sure they have correctly diagnosed the problem. They say it is fixed with a new libjpeg-turbo in a Pillow release due out January 2.

comment:14 by Tom Goddard, 22 months ago

Only ARM CPU versions of PIL on Mac have this bug, Intel works correctly.

The jpeg corruption does not occur on Mac Intel ChimeraX 1.7 with PIL 10.0.1. That PIL has libjpeg.62.4.0.dylib and libopenjp2.2.5.0.dylib. Tested on descartes.cgl.ucsf.edu. The 169.jpg attached to this ticket is reported as corrupted on Intel Mac by ImageMagick identify.

Last edited 22 months ago by Tom Goddard (previous) (diff)

comment:15 by Tom Goddard, 22 months ago

Also ChimeraX 1.7 on Windows 11 does not create corrupt jpegs. Tested on vizvault.cgl.ucsf.edu.

So it looks like the problem is only on Mac ARM computers.

comment:16 by Tom Goddard, 22 months ago

I reported this Pillow bug on the Pillow Github site with a test case

https://github.com/python-pillow/Pillow/issues/7628

comment:17 by Tom Goddard, 22 months ago

Pillow developer Andrew Murray reports on the Pillow Github issue I made that he tested the Pillow to be released Jan 2 with my test case and it works correctly. We can update Pillow when that release is made.

comment:18 by Tom Goddard, 22 months ago

Resolution: fixed
Status: assignedclosed

Fixed in 1.7.1 and 1.8 daily builds.

Pillow 10.2.0 fixed the problem writing corrupted images and I have updated ChimeraX to use that version.

Note: See TracTickets for help on using tickets.