Linux OpenBLAS library threading problems

[Tristan Croll]

This came up a while back, and I have the below workaround defined in my chimerax_start directory. It seems that libopenblas (used by Numpy for various linear algebra tasks) has a bug in its multi-threaded implementation. By default it sets the number of threads to the number of available CPUs, but for me (on multiple machines in Fedora or CentOS), any number of threads more than one leads to 200% CPU usage when I do something as simple as:

import numpy
def invert(*_):
    numpy.linalg.invert(numpy.identity(3))
session.triggers.add_handler('new frame', invert)

Applying the following drops CPU usage back to 4-5% on my machines:

import ctypes
openblas_lib = ctypes.cdll.LoadLibrary('/home/tic20/apps/chimerax/lib/python3.6/site-packages/numpy/.libs/libopenblasp-r0-39a31c03.2.18.so')
openblas_lib.openblas_set_num_threads(1)

Until OpenBLAS is fixed, I'd suggest doing this by default in Linux builds.

[Conrad Huang]

Hi, Tristan.

Is this still a problem? If so, I will put the workaround into the ChimeraX start-up code for Linux.

Conrad

[Tristan Croll]

Sorry for letting this drop off the radar, but it turns out it's still an issue. Using the following test in ChimeraX:

m = session.models.list()[0]

def invert_position(*_, model = m):
    m.scene_position.inverse()
    #p = m.scene_position

h = session.triggers.add_handler('new frame', invert_position)

... then:

OPENBLAS_NUM_THREADS=1 ChimeraX 1igr
CPU usage = 5.0%

OPENBLAS_NUM_THREADS=2 ChimeraX 1igr
CPU usage = 105%

OPENBLAS_NUM_THREADS=3 ChimeraX 1igr
CPU usage = 202%

... and larger thread counts also lead to ~200% CPU usage. I gather the same effect as the above can be had by setting os.environ['OPENBLAS_NUM_THREADS'] *before* numpy is imported for the first time.

By the way, the problem gets horribly, horribly worse for bigger matrices. If I instead do:

import numpy
def invert_matrix(*_):
    numpy.linalg.inv(numpy.identity(50))

h = session.triggers.add_handler('new frame', invert_matrix)

Then (if ChimeraX is run without limiting thread count) CPU usage spikes to 3100%! Yet if I limit it to one thread, CPU usage remains at 5% and the display is smooth.

[Conrad Huang]

Applied workaround in 5f7cc9259.

Should be in daily build tomorrow.