Session save fails for numpy array of strings

[Tristan Croll]

The following bug report has been submitted:
Platform:        Linux-3.10.0-957.12.2.el7.x86_64-x86_64-with-centos-7.6.1810-Core
ChimeraX Version: 0.91 (2019-12-03)
Description
This is a little weird... as per #2589, last night saving a session with a `take_snapshot()` method that includes a Numpy array of strings was raising an exception about not being able to create a 1000-odd dimensional array. This morning I added the follwing at line 203 of state.py:

{{{
                items = [_copy(o) for o in data]
            print(data.__class__)
            return data.__class__(items)
}}}

... to get a better understanding of what's going on. For reasons unfathomable to me, that changed the traceback to this (much more informative) one - even after I reverted state.py to remove the `print` call. Anyway, the root cause of the traceback is that the session save code doesn't know what to do with a Numpy array of strings - if I convert it to a list, everything is fine.

Log:
Startup Messages  
---  
warning | 'clip' is a prefix of an existing command 'clipper'  
  
UCSF ChimeraX version: 0.91 (2019-12-03)  
© 2016-2019 Regents of the University of California. All rights reserved.  
How to cite UCSF ChimeraX  
Successfully installed
'ChimeraX_ISOLDE-1.0b3.dev9-cp37-cp37m-linux_x86_64.whl'  
Looking in indexes: https://pypi.org/simple,
https://cxtoolshed.rbvi.ucsf.edu/pypi/  
Processing
/home/tic20/.cache/ChimeraX/0.91/installers/ChimeraX_ISOLDE-1.0b3.dev9-cp37-cp37m-linux_x86_64.whl  
Requirement already satisfied, skipping upgrade: ChimeraX-Clipper>=0.9.13 in
/home/tic20/.local/share/ChimeraX/0.91/site-packages (from ChimeraX-
ISOLDE==1.0b3.dev9) (0.9.13)  
Requirement already satisfied, skipping upgrade: ChimeraX-Atomic>=1.0 in
/opt/UCSF/ChimeraX-daily/lib/python3.7/site-packages (from ChimeraX-
ISOLDE==1.0b3.dev9) (1.0)  
Requirement already satisfied, skipping upgrade: ChimeraX-Core==0.91 in
/opt/UCSF/ChimeraX-daily/lib/python3.7/site-packages (from ChimeraX-
ISOLDE==1.0b3.dev9) (0.91)  
Installing collected packages: ChimeraX-ISOLDE  
Found existing installation: ChimeraX-ISOLDE 1.0b3.dev9  
Uninstalling ChimeraX-ISOLDE-1.0b3.dev9:  
Successfully uninstalled ChimeraX-ISOLDE-1.0b3.dev9  
Successfully installed ChimeraX-ISOLDE-1.0b3.dev9  
Lock 140266156042576 acquired on
/home/tic20/.cache/ChimeraX/0.91/toolshed/bundle_info.cache.lock  
Lock 140266156042576 released on
/home/tic20/.cache/ChimeraX/0.91/toolshed/bundle_info.cache.lock  
  

> open test_save_restore.py

> open 3io0 structureFactors true

Summary of feedback from opening 3io0 fetched from pdb  
---  
warning | WARNING: multiple experimental reflection datasets found:  
F_meas_au, F_meas_sigma_au,  
pdbx_F_plus, pdbx_F_plus_sigma, pdbx_F_minus, pdbx_F_minus_sigma,  
pdbx_anom_difference, pdbx_anom_difference_sigma,  
intensity_meas, intensity_sigma,  
pdbx_I_plus, pdbx_I_plus_sigma, pdbx_I_minus, pdbx_I_minus_sigma  
Automatically choosing "intensity_meas, intensity_sigma".  
note | Reflection data provided as intensities. Performing French & Wilson
scaling to convert to amplitudes...  
  
3io0 title:  
Crystal structure of EtuB from Clostridium kluyveri [more info...]  
  
Chain information for 3io0  
---  
Chain | Description  
1.1/A | EtuB protein  
  
3io0 mmCIF Assemblies  
---  
1| author_and_software_defined_assembly  
  
  

> isolde restrain distances #1

> addh

Summary of feedback from adding hydrogens to 3io0 #1.1  
---  
notes | Termini for 3io0 (#1.1) chain A determined from SEQRES records  
Chain-initial residues that are actual N termini:  
Chain-initial residues that are not actual N termini: /A PRO 76  
Chain-final residues that are actual C termini: /A PHE 304  
Chain-final residues that are not actual C termini:  
Missing OXT added to C-terminal residue /A PHE 304  
198 hydrogen bonds  
1692 hydrogens added  
  

> isolde start

> set selectionWidth 4

executed test_save_restore.py  
Done loading forcefield  

> select clear

> save test.cxs

Session saving is not yet implemented for ISOLDE. Custom restraints will not
be saved.  

Traceback (most recent call last):  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/core/session.py", line 796, in save  
session.save(output, version=version, include_maps=include_maps)  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/core/session.py", line 556, in save  
mgr.discovery(self._state_containers)  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/core/session.py", line 255, in discovery  
self.processed[key] = self.process(obj, parents)  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/core/session.py", line 288, in process  
return copy_state(data, convert=convert)  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/core/state.py", line 208, in copy_state  
return _copy(data)  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/core/state.py", line 195, in _copy  
items = [(_copy(k), _copy(v)) for k, v in data.items()]  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/core/state.py", line 195, in <listcomp>  
items = [(_copy(k), _copy(v)) for k, v in data.items()]  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/core/state.py", line 203, in _copy  
return data.__class__(items)  
TypeError: 'str' object cannot be interpreted as an integer  
  
TypeError: 'str' object cannot be interpreted as an integer  
  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/core/state.py", line 203, in _copy  
return data.__class__(items)  
  
See log for complete Python traceback.  
  
Traceback (most recent call last):  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/cmd_line/tool.py", line 258, in execute  
cmd.run(cmd_text)  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/core/commands/cli.py", line 2837, in run  
result = ci.function(session, **kw_args)  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/core/commands/save.py", line 64, in save  
_handle_unexpected_keyword_error(e, 5)  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/core/commands/save.py", line 61, in save  
fmt.export(session, filename, fmt.nicknames[0], **kw)  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/core/io.py", line 220, in export  
result = self.export_func(session, path, **kw)  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/core/session.py", line 796, in save  
session.save(output, version=version, include_maps=include_maps)  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/core/session.py", line 556, in save  
mgr.discovery(self._state_containers)  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/core/session.py", line 255, in discovery  
self.processed[key] = self.process(obj, parents)  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/core/session.py", line 288, in process  
return copy_state(data, convert=convert)  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/core/state.py", line 208, in copy_state  
return _copy(data)  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/core/state.py", line 195, in _copy  
items = [(_copy(k), _copy(v)) for k, v in data.items()]  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/core/state.py", line 195, in <listcomp>  
items = [(_copy(k), _copy(v)) for k, v in data.items()]  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/core/state.py", line 203, in _copy  
return data.__class__(items)  
TypeError: 'str' object cannot be interpreted as an integer  
  
TypeError: 'str' object cannot be interpreted as an integer  
  
File "/opt/UCSF/ChimeraX-daily/lib/python3.7/site-
packages/chimerax/core/state.py", line 203, in _copy  
return data.__class__(items)  
  
See log for complete Python traceback.  
  




OpenGL version: 3.3.0 NVIDIA 418.87.01
OpenGL renderer: TITAN Xp/PCIe/SSE2
OpenGL vendor: NVIDIA Corporation

[Greg Couch]

I've committed a fix. Please let me know if you are able to confirm if it works. If you want to cherry pick the fix, it is commit 580eba8117f50b313a57e7209efa9ab8822f4528.

[Tristan Croll]

Afraid not. New traceback is below.


Traceback (most recent call last):
   File 
"/opt/UCSF/ChimeraX-daily/lib/python3.7/site-packages/chimerax/core/session.py", 
line 796, in save
     session.save(output, version=version, include_maps=include_maps)
   File 
"/opt/UCSF/ChimeraX-daily/lib/python3.7/site-packages/chimerax/core/session.py", 
line 561, in save
     fserialize(stream, data)
   File 
"/opt/UCSF/ChimeraX-daily/lib/python3.7/site-packages/chimerax/core/serialize.py", 
line 63, in msgpack_serialize
     stream.write(packer.pack(obj))
   File "msgpack/_packer.pyx", line 284, in msgpack._cmsgpack.Packer.pack
   File "msgpack/_packer.pyx", line 290, in msgpack._cmsgpack.Packer.pack
   File "msgpack/_packer.pyx", line 287, in msgpack._cmsgpack.Packer.pack
   File "msgpack/_packer.pyx", line 234, in 
msgpack._cmsgpack.Packer._pack
   File "msgpack/_packer.pyx", line 277, in 
msgpack._cmsgpack.Packer._pack
   File "stringsource", line 67, in 
cfunc.to_py.__Pyx_CFunc_object____object___to_py.wrap
   File "_serialize.pyx", line 215, in core._serialize._encode_ext
   File "_serialize.pyx", line 140, in core._serialize._encode_ndarray
TypeError: Can not serialize numpy arrays of objects

TypeError: Can not serialize numpy arrays of objects

File "_serialize.pyx", line 140, in core._serialize._encode_ndarray

See log for complete Python traceback.

Traceback (most recent call last):
   File 
"/opt/UCSF/ChimeraX-daily/lib/python3.7/site-packages/chimerax/cmd_line/tool.py", 
line 258, in execute
     cmd.run(cmd_text)
   File 
"/opt/UCSF/ChimeraX-daily/lib/python3.7/site-packages/chimerax/core/commands/cli.py", 
line 2837, in run
     result = ci.function(session, **kw_args)
   File 
"/opt/UCSF/ChimeraX-daily/lib/python3.7/site-packages/chimerax/core/commands/save.py", 
line 64, in save
     _handle_unexpected_keyword_error(e, 5)
   File 
"/opt/UCSF/ChimeraX-daily/lib/python3.7/site-packages/chimerax/core/commands/save.py", 
line 61, in save
     fmt.export(session, filename, fmt.nicknames[0], **kw)
   File 
"/opt/UCSF/ChimeraX-daily/lib/python3.7/site-packages/chimerax/core/io.py", 
line 225, in export
     result = self.export_func(session, path, **kw)
   File 
"/opt/UCSF/ChimeraX-daily/lib/python3.7/site-packages/chimerax/core/session.py", 
line 796, in save
     session.save(output, version=version, include_maps=include_maps)
   File 
"/opt/UCSF/ChimeraX-daily/lib/python3.7/site-packages/chimerax/core/session.py", 
line 561, in save
     fserialize(stream, data)
   File 
"/opt/UCSF/ChimeraX-daily/lib/python3.7/site-packages/chimerax/core/serialize.py", 
line 63, in msgpack_serialize
     stream.write(packer.pack(obj))
   File "msgpack/_packer.pyx", line 284, in msgpack._cmsgpack.Packer.pack
   File "msgpack/_packer.pyx", line 290, in msgpack._cmsgpack.Packer.pack
   File "msgpack/_packer.pyx", line 287, in msgpack._cmsgpack.Packer.pack
   File "msgpack/_packer.pyx", line 234, in 
msgpack._cmsgpack.Packer._pack
   File "msgpack/_packer.pyx", line 277, in 
msgpack._cmsgpack.Packer._pack
   File "stringsource", line 67, in 
cfunc.to_py.__Pyx_CFunc_object____object___to_py.wrap
   File "_serialize.pyx", line 215, in core._serialize._encode_ext
   File "_serialize.pyx", line 140, in core._serialize._encode_ndarray
TypeError: Can not serialize numpy arrays of objects

TypeError: Can not serialize numpy arrays of objects

File "_serialize.pyx", line 140, in core._serialize._encode_ndarray

See log for complete Python traceback.

On 2019-12-09 23:10, ChimeraX wrote:

[Greg Couch]

That's the error I expected :-).

The limitation comes from trying to support numpy structured arrays. If there non-numerical types, ie., objects, in the structured array, then it can't be written as a simple byte stream (strings are considered numbers sometimes, don't know the numpy rules). And serializing each element separately when they are numerical is expensive and wasteful. Unless the array is multidimensional, or you want to use numpy operations, a numpy array of objects would be better as a Python list. That said, it would be relatively easy to support simple arrays of objects by introducing another serialization primitive. Would that be sufficient?

[Tristan Croll]

Honestly, in this case there’s no harm at all in saving it as a list (since restoring loops over it in Python anyway, there’s no performance benefit). Mainly just reported it to do my civic duty. :)
 

[Greg Couch]

Changed traceback to say "Serializing numpy arrays of objects is not supported" to show that it is intentional.