Opened 5 years ago

Last modified 3 years ago

#3460 assigned defect

Atom-spec parser slow and possibly leaks memory

Reported by: Eric Pettersen Owned by: Eric Pettersen
Priority: moderate Milestone:
Component: Performance Version:
Keywords: Cc: chimera-programmers
Blocked By: Blocking:
Notify when closed: Platform: all
Project: ChimeraX

Description

From chimera-users list:

I'm testing driving ChimeraX (or Chimera) from Jalview, in particular colouring structures using the Jalview alignment residue colouring.
Ideally this should perform well at scale for large numbers of structures and/or colours (for example a graduated colouring based on some property).
I've noticed that ChimeraX appears to do this slower than Chimera.
Is there a reason, or something I should do to make the command faster?

Here's an example like for like comparison using the Chimera/X command line (also attached as a text file)
[for this ticket, just provided as an attachment rather than inline]

Attachments (2)

commands.txt (17.5 KB ) - added by Tom Goddard 5 years ago.
atomspec.lalr (10.8 KB ) - added by Eric Pettersen 3 years ago.

Download all attachments as: .zip

Change History (24)

comment:1 by Eric Pettersen, 5 years ago

Did a lot of profiling, and the atom-spec parser takes about 5 seconds to process the color commands, even with no structures open. Also, repeating the command leaks significant memory each time (30-ish MB) and after about 4 repetitions the execution gets slower by about .5 seconds each time, again with no structures open.

comment:2 by Tom Goddard, 5 years ago

Status and logging are also making running these 383 color commands (one per residue) slow. On my MacBook Pro coloring is done in 6.5 seconds. Commenting out status messages (in MainWindow.status) and the time is 3.0 seconds. Status messages are limited to 60 per-second by the OpenGL refresh rate and each color command is giving a status message. Each of the 383 commands also gets individually logged. Commenting out command logging (in core/commands/cli.py Command.run()) reduces the time from 3.0 to 2.7 seconds.

comment:3 by Tom Goddard, 5 years ago

Original email that led to this ticket.

Begin forwarded message:

From: "Mungo Carstairs (Staff)"
Subject: [chimerax-users] Is color command slower in ChimeraX than Chimera?
Date: June 30, 2020 at 2:14:27 AM PDT
To: "chimerax-users@…" <chimerax-users@…>

Hi,

I'm testing driving ChimeraX (or Chimera) from Jalview, in particular colouring structures using the Jalview alignment residue colouring.
Ideally this should perform well at scale for large numbers of structures and/or colours (for example a graduated colouring based on some property).
I've noticed that ChimeraX appears to do this slower than Chimera.
Is there a reason, or something I should do to make the command faster?

Here's an example like for like comparison using the Chimera/X command line (also attached as a text file):

Chimera 1.15 (build 42156)

open cifID:4zho cifID:5h92 cifID:1a70 cifID:4zhp
focus
color #977b64 #0:2.A,2.B;color #e46daf #0:3.A,3.B;color #7dc891 #0:4.A,4.B;color #768daf #0:5.A,5.B;color #bb1691 #0:6.A,6.B;color #1c5355 #0:7.A,7.B;color #5b1873 #0:8.A,8.B;color #d0af91 #0:9.A,9.B;color #6cb737 #0:10.A,10.B;color #8d7c55 #0:11.A,11.B;color #844173 #0:12.A,12.B;color #1f15a0 #0:13.A,13.B;color #a27f91 #0:14.A,14.B;color #19cb64 #0:15.A,15.B;color #484537 #0:16.A,16.B;color #61be37 #0:17.A,17.B;color #84a173 #0:18.A,18.B;color #c76691 #0:19.A,19.B;color #8f31a0 #0:20.A,20.B;color #9eb9af #0:21.A,21.B;color #531564 #0:22.A,22.B;color #707046 #0:23.A,23.B;color #9b80af #0:24.A,24.B;color #ca18a0 #0:25.A,25.B;color #377382 #0:26.A,26.B;color #26a137 #0:27.A,27.B;color #d3b1a0 #0:28.A,28.B;color #e267af #0:29.A,29.B;color #c023af #0:30.A,30.B;color #b18da0 #0:31.A,31.B;color #88bb55 #0:32.A,32.B;color #1fada0 #0:33.A,33.B;color #b22773 #0:34.A,34.B;color #ad19be #0:35.A,35.B;color #6483af #0:36.A,36.B;color #c9de91 #0:37.A,37.B;color #aab2be #0:38.A,38.B;color #59d164 #0:39.A,39.B;color #6d6037 #0:40.A,40.B;color #1cd937 #0:41.A,41.B;color #b5cb82 #0:42.A,42.B;color #e28191 #0:43.A,43.B;color #9537a0 #0:44.A,44.B;color #d6b0a0 #0:45.A,45.B;color #3bcf46 #0:46.A,46.B;color #2c6737 #0:47.A,47.B;color #e586af #0:48.A,48.B;color #e22daf #0:49.A,49.B;color #e29d91 #0:50.A,50.B;color #493582 #0:51.A,51.B;color #cacca0 #0:52.A,52.B;color #7f2855 #0:53.A,53.B;color #86dd91 #0:54.A,54.B;color #b55764 #0:55.A,55.B;color #4ac155 #0:56.A,56.B;color #614a37 #0:57.A,57.B;color #9e69af #0:58.A,58.B;color #5f8855 #0:59.A,59.B;color #958fa0 #0:60.A,60.B;color #9e45af #0:61.A,61.B;color #b9beaf #0:62.A,62.B;color #4add55 #0:63.A,63.B;color #5ba273 #0:64.A,64.B;color #bedf37 #0:65.A,65.B;color #359546 #0:66.A,66.B;color #262d37 #0:67.A,67.B;color #cf5bbe #0:68.A,68.B;color #505c46 #0:69.A,69.B;color #76db37 #0:70.A,70.B;color #3d37be #0:71.A,71.B;color #a28391 #0:72.A,72.B;color #b5cf64 #0:73.A,73.B;color #88b2a0 #0:74.A,74.B;color #bb4a91 #0:75.A,75.B;color #dc8755 #0:76.A,76.B;color #d579a0 #0:77.A,77.B;color #7c9846 #0:78.A,78.B;color #b57b82 #0:79.A,79.B;color #79cd46 #0:80.A,80.B;color #aa1abe #0:81.A,81.B;color #ab4873 #0:82.A,82.B;color #c6df91 #0:83.A,83.B;color #b3b3be #0:84.A,84.B;color #802d91 #0:85.A,85.B;color #d9c4af #0:86.A,86.B;color #c64d91 #0:87.A,87.B;color #857b46 #0:88.A,88.B;color #a26d91 #0:89.A,89.B;color #c62991 #0:90.A,90.B;color #adb1a0 #0:91.A,91.B;color #584991 #0:92.A,92.B;color #93b382 #0:93.A,93.B;color #48e137 #0:94.A,94.B;color #88c56f #1:1.C;color #7b5d60 #1:2.C;color #32d660 #1:3.C;color #339b7e #1:4.C;color #702460 #1:5.C;color #6725ba #1:6.C;color #dc536f #1:7.C;color #a3db7e #1:8.C;color #a8918d #1:9.C;color #d84eba #1:10.C;color #758b7e #1:11.C;color #88d751 #1:12.C;color #75ab7e #1:13.C;color #8e527e #1:14.C;color #cf53d8 #1:15.C;color #dc636f #1:16.C;color #7d288d #1:17.C;color #707460 #1:18.C;color #6fde51 #1:19.C;color #35a960 #1:20.C;color #627d8d #1:21.C;color #bb42ab #1:22.C;color #327060 #1:23.C;color #a3357e #1:24.C;color #28bd8d #1:25.C;color #8f91ba #1:26.C;color #c856d8 #1:27.C;color #795760 #1:28.C;color #c067c9 #1:29.C;color #669b6f #1:30.C;color #adabd8 #1:31.C;color #889d51 #1:32.C;color #c18f9c #1:33.C;color #26bd51 #1:34.C;color #55cdba #1:35.C;color #7e1a60 #1:36.C;color #238451 #1:37.C;color #68678d #1:38.C;color #6da451 #1:39.C;color #a315d8 #1:40.C;color #88256f #1:41.C;color #1f17ba #1:42.C;color #4a456f #1:43.C;color #6da051 #1:44.C;color #a4bfc9 #1:45.C;color #9557ba #1:46.C;color #d6d0ba #1:47.C;color #791d60 #1:48.C;color #e2e1ab #1:49.C;color #1c616f #1:50.C;color #7fda6f #1:51.C;color #8e907e #1:52.C;color #77559c #1:53.C;color #c4bf8d #1:54.C;color #77759c #1:55.C;color #1658d8 #1:56.C;color #add1d8 #1:57.C;color #aa5aba #1:58.C;color #95d3ba #1:59.C;color #adadd8 #1:60.C;color #aa36ba #1:61.C;color #95afba #1:62.C;color #aa74d8 #1:63.C;color #731bd8 #1:64.C;color #442b6f #1:65.C;color #8f1dba #1:66.C;color #882d51 #1:67.C;color #9b2eab #1:68.C;color #764d51 #1:69.C;color #6adb51 #1:70.C;color #359160 #1:71.C;color #e283ab #1:72.C;color #ddb160 #1:73.C;color #cab2ba #1:74.C;color #5559ba #1:75.C;color #8a876f #1:76.C;color #c76aab #1:77.C;color #62a76f #1:78.C;color #793f60 #1:79.C;color #23be51 #1:80.C;color #bab09c #1:81.C;color #99397e #1:82.C;color #a42bc9 #1:83.C;color #53597e #1:84.C;color #165ad8 #1:85.C;color #9d888d #1:86.C;color #d04dab #1:87.C;color #b1d5ba #1:88.C;color #c66dab #1:89.C;color #44a151 #1:90.C;color #df5760 #1:91.C;color #66df6f #1:92.C;color #cf1dd8 #1:93.C;color #a269ab #1:94.C;color #551fba #1:95.C;color #e4a7c9 #1:96.C;color #1f5c8b #2:1.A;color #a68a40 #2:2.A;color #5f3140 #2:3.A;color #76147c #2:4.A;color #9d5140 #2:5.A;color #6c8ed6 #2:6.A;color #37804f #2:7.A;color #8edb31 #2:8.A;color #7b64e5 #2:9.A;color #ab2140 #2:10.A;color #a89a40 #2:11.A;color #b53231 #2:12.A;color #7e247c #2:13.A;color #6125d6 #2:14.A;color #a2265e #2:15.A;color #97bd7c #2:16.A;color #aa556d #2:17.A;color #a9a140 #2:18.A;color #9c3931 #2:19.A;color #9e407c #2:20.A;color #8faa6d #2:21.A;color #166f8b #2:22.A;color #9bd97c #2:23.A;color #76dad6 #2:24.A;color #cd90e5 #2:25.A;color #2628d6 #2:26.A;color #97ce31 #2:27.A;color #e2c07c #2:28.A;color #933a4f #2:29.A;color #cf328b #2:30.A;color #a27e5e #2:31.A;color #b5ca31 #2:32.A;color #946222 #2:33.A;color #8f546d #2:34.A;color #28a040 #2:35.A;color #ab4740 #2:36.A;color #8c1b6d #2:37.A;color #95946d #2:38.A;color #d63b6d #2:39.A;color #76ba5e #2:40.A;color #b5524f #2:41.A;color #8880d6 #2:42.A;color #b3ae8b #2:43.A;color #9acd31 #2:44.A;color #77924f #2:45.A;color #2cc0d6 #2:46.A;color #a9a340 #2:47.A;color #e2867c #2:48.A;color #b5b431 #2:49.A;color #498e4f #2:50.A;color #64627c #2:51.A;color #85188b #2:52.A;color #e4827c #2:53.A;color #1f1a6d #2:54.A;color #a4a27c #2:55.A;color #cdb2e5 #2:56.A;color #60a45e #2:57.A;color #7d2d40 #2:58.A;color #80c45e #2:59.A;color #444422 #2:60.A;color #7ddb40 #2:61.A;color #688240 #2:62.A;color #1fbfd6 #2:63.A;color #46c05e #2:64.A;color #71584f #2:65.A;color #2686d6 #2:66.A;color #353ce5 #2:67.A;color #c85b8b #2:68.A;color #a37a31 #2:69.A;color #3518e5 #2:70.A;color #4882d6 #2:71.A;color #b55631 #2:72.A;color #74487c #2:73.A;color #6149d6 #2:74.A;color #282c40 #2:75.A;color #211e8b #2:76.A;color #22978b #2:77.A;color #7998e5 #2:78.A;color #a66c40 #2:79.A;color #501931 #2:80.A;color #15dd7c #2:81.A;color #ca756d #2:82.A;color #3b94e5 #2:83.A;color #262cd6 #2:84.A;color #7fc322 #2:85.A;color #a88840 #2:86.A;color #2b7a8b #2:87.A;color #486cd6 #2:88.A;color #994031 #2:89.A;color #71ce31 #2:90.A;color #3a8440 #2:91.A;color #ffffff #2:92.A;color #a2c25e #2:93.A;color #cf968b #2:94.A;color #28c440 #2:95.A;color #7b3ee5 #2:96.A;color #6aa8d6 #2:97.A;color #977b82 #3:2.A;color #683191 #3:3.A;color #7dc8af #3:4.A;color #768dcd #3:5.A;color #bb16af #3:6.A;color #1c5373 #3:7.A;color #3727a0 #3:8.A;color #d0afaf #3:9.A;color #6cb755 #3:10.A;color #8d7c73 #3:11.A;color #26d755 #3:12.A;color #1f15be #3:13.A;color #7e3464 #3:14.A;color #df35be #3:15.A;color #484555 #3:16.A;color #9764cd #3:17.A;color #84a191 #3:18.A;color #c766af #3:19.A;color #6f9a55 #3:20.A;color #9eb9cd #3:21.A;color #531582 #3:22.A;color #707064 #3:23.A;color #9b80cd #3:24.A;color #d0dbaf #3:25.A;color #3773a0 #3:26.A;color #26a155 #3:27.A;color #414855 #3:28.A;color #e267cd #3:29.A;color #c023cd #3:30.A;color #b18dbe #3:31.A;color #84d991 #3:32.A;color #1fadbe #3:33.A;color #bc63cd #3:34.A;color #ad19dc #3:35.A;color #6483cd #3:36.A;color #c9deaf #3:37.A;color #aab2dc #3:38.A;color #59d182 #3:39.A;color #6d6055 #3:40.A;color #1cd955 #3:41.A;color #b5cba0 #3:42.A;color #e281af #3:43.A;color #9537be #3:44.A;color #d6b0be #3:45.A;color #3bcf64 #3:46.A;color #2c6755 #3:47.A;color #e586cd #3:48.A;color #e22dcd #3:49.A;color #e29daf #3:50.A;color #4935a0 #3:51.A;color #caccbe #3:52.A;color #d573be #3:53.A;color #e229cd #3:54.A;color #b55782 #3:55.A;color #e449cd #3:56.A;color #614a55 #3:57.A;color #9e69cd #3:58.A;color #5f8873 #3:59.A;color #958fbe #3:60.A;color #9e45cd #3:61.A;color #975564 #3:62.A;color #1cdd73 #3:63.A;color #5ba291 #3:64.A;color #bedf55 #3:65.A;color #359564 #3:66.A;color #262d55 #3:67.A;color #711fa0 #3:68.A;color #505c64 #3:69.A;color #662773 #3:70.A;color #d319be #3:71.A;color #e2a1cd #3:72.A;color #b5cf82 #3:73.A;color #88b2be #3:74.A;color #bb4aaf #3:75.A;color #dc8773 #3:76.A;color #d579be #3:77.A;color #7c9864 #3:78.A;color #9d6c91 #3:79.A;color #79cd64 #3:80.A;color #aa1adc #3:81.A;color #ab4891 #3:82.A;color #1ca373 #3:83.A;color #8b4a73 #3:84.A;color #802daf #3:85.A;color #d9c4cd #3:86.A;color #c64daf #3:87.A;color #857b64 #3:88.A;color #a26daf #3:89.A;color #c629af #3:90.A;color #adb1be #3:91.A;color #5849af #3:92.A;color #93b3a0 #3:93.A;color #48e155 #3:94.A;color #93d3a0 #3:95.A;color #dc4d73 #3:96.A;color #d53fbe #3:97.A;color #6a4f55 #3:98.A

On my Macbook these color commands take about 2 seconds.
Email signature
ChimeraX 1.0 (2020-06-04):

open pdb:4zho pdb:5h92 pdb:1a70 pdb:4zhp
view
color #1/A:2/B:2 #977b64;color #1/A:3/B:3 #e46daf;color #1/A:4/B:4 #7dc891;color #1/A:5/B:5 #768daf;color #1/A:6/B:6 #bb1691;color #1/A:7/B:7 #1c5355;color #1/A:8/B:8 #5b1873;color #1/A:9/B:9 #d0af91;color #1/A:10/B:10 #6cb737;color #1/A:11/B:11 #8d7c55;color #1/A:12/B:12 #844173;color #1/A:13/B:13 #1f15a0;color #1/A:14/B:14 #a27f91;color #1/A:15/B:15 #19cb64;color #1/A:16/B:16 #484537;color #1/A:17/B:17 #61be37;color #1/A:18/B:18 #84a173;color #1/A:19/B:19 #c76691;color #1/A:20/B:20 #8f31a0;color #1/A:21/B:21 #9eb9af;color #1/A:22/B:22 #531564;color #1/A:23/B:23 #707046;color #1/A:24/B:24 #9b80af;color #1/A:25/B:25 #ca18a0;color #1/A:26/B:26 #377382;color #1/A:27/B:27 #26a137;color #1/A:28/B:28 #d3b1a0;color #1/A:29/B:29 #e267af;color #1/A:30/B:30 #c023af;color #1/A:31/B:31 #b18da0;color #1/A:32/B:32 #88bb55;color #1/A:33/B:33 #1fada0;color #1/A:34/B:34 #b22773;color #1/A:35/B:35 #ad19be;color #1/A:36/B:36 #6483af;color #1/A:37/B:37 #c9de91;color #1/A:38/B:38 #aab2be;color #1/A:39/B:39 #59d164;color #1/A:40/B:40 #6d6037;color #1/A:41/B:41 #1cd937;color #1/A:42/B:42 #b5cb82;color #1/A:43/B:43 #e28191;color #1/A:44/B:44 #9537a0;color #1/A:45/B:45 #d6b0a0;color #1/A:46/B:46 #3bcf46;color #1/A:47/B:47 #2c6737;color #1/A:48/B:48 #e586af;color #1/A:49/B:49 #e22daf;color #1/A:50/B:50 #e29d91;color #1/A:51/B:51 #493582;color #1/A:52/B:52 #cacca0;color #1/A:53/B:53 #7f2855;color #1/A:54/B:54 #86dd91;color #1/A:55/B:55 #b55764;color #1/A:56/B:56 #4ac155;color #1/A:57/B:57 #614a37;color #1/A:58/B:58 #9e69af;color #1/A:59/B:59 #5f8855;color #1/A:60/B:60 #958fa0;color #1/A:61/B:61 #9e45af;color #1/A:62/B:62 #b9beaf;color #1/A:63/B:63 #4add55;color #1/A:64/B:64 #5ba273;color #1/A:65/B:65 #bedf37;color #1/A:66/B:66 #359546;color #1/A:67/B:67 #262d37;color #1/A:68/B:68 #cf5bbe;color #1/A:69/B:69 #505c46;color #1/A:70/B:70 #76db37;color #1/A:71/B:71 #3d37be;color #1/A:72/B:72 #a28391;color #1/A:73/B:73 #b5cf64;color #1/A:74/B:74 #88b2a0;color #1/A:75/B:75 #bb4a91;color #1/A:76/B:76 #dc8755;color #1/A:77/B:77 #d579a0;color #1/A:78/B:78 #7c9846;color #1/A:79/B:79 #b57b82;color #1/A:80/B:80 #79cd46;color #1/A:81/B:81 #aa1abe;color #1/A:82/B:82 #ab4873;color #1/A:83/B:83 #c6df91;color #1/A:84/B:84 #b3b3be;color #1/A:85/B:85 #802d91;color #1/A:86/B:86 #d9c4af;color #1/A:87/B:87 #c64d91;color #1/A:88/B:88 #857b46;color #1/A:89/B:89 #a26d91;color #1/A:90/B:90 #c62991;color #1/A:91/B:91 #adb1a0;color #1/A:92/B:92 #584991;color #1/A:93/B:93 #93b382;color #1/A:94/B:94 #48e137;color #2/C:1 #88c56f;color #2/C:2 #7b5d60;color #2/C:3 #32d660;color #2/C:4 #339b7e;color #2/C:5 #702460;color #2/C:6 #6725ba;color #2/C:7 #dc536f;color #2/C:8 #a3db7e;color #2/C:9 #a8918d;color #2/C:10 #d84eba;color #2/C:11 #758b7e;color #2/C:12 #88d751;color #2/C:13 #75ab7e;color #2/C:14 #8e527e;color #2/C:15 #cf53d8;color #2/C:16 #dc636f;color #2/C:17 #7d288d;color #2/C:18 #707460;color #2/C:19 #6fde51;color #2/C:20 #35a960;color #2/C:21 #627d8d;color #2/C:22 #bb42ab;color #2/C:23 #327060;color #2/C:24 #a3357e;color #2/C:25 #28bd8d;color #2/C:26 #8f91ba;color #2/C:27 #c856d8;color #2/C:28 #795760;color #2/C:29 #c067c9;color #2/C:30 #669b6f;color #2/C:31 #adabd8;color #2/C:32 #889d51;color #2/C:33 #c18f9c;color #2/C:34 #26bd51;color #2/C:35 #55cdba;color #2/C:36 #7e1a60;color #2/C:37 #238451;color #2/C:38 #68678d;color #2/C:39 #6da451;color #2/C:40 #a315d8;color #2/C:41 #88256f;color #2/C:42 #1f17ba;color #2/C:43 #4a456f;color #2/C:44 #6da051;color #2/C:45 #a4bfc9;color #2/C:46 #9557ba;color #2/C:47 #d6d0ba;color #2/C:48 #791d60;color #2/C:49 #e2e1ab;color #2/C:50 #1c616f;color #2/C:51 #7fda6f;color #2/C:52 #8e907e;color #2/C:53 #77559c;color #2/C:54 #c4bf8d;color #2/C:55 #77759c;color #2/C:56 #1658d8;color #2/C:57 #add1d8;color #2/C:58 #aa5aba;color #2/C:59 #95d3ba;color #2/C:60 #adadd8;color #2/C:61 #aa36ba;color #2/C:62 #95afba;color #2/C:63 #aa74d8;color #2/C:64 #731bd8;color #2/C:65 #442b6f;color #2/C:66 #8f1dba;color #2/C:67 #882d51;color #2/C:68 #9b2eab;color #2/C:69 #764d51;color #2/C:70 #6adb51;color #2/C:71 #359160;color #2/C:72 #e283ab;color #2/C:73 #ddb160;color #2/C:74 #cab2ba;color #2/C:75 #5559ba;color #2/C:76 #8a876f;color #2/C:77 #c76aab;color #2/C:78 #62a76f;color #2/C:79 #793f60;color #2/C:80 #23be51;color #2/C:81 #bab09c;color #2/C:82 #99397e;color #2/C:83 #a42bc9;color #2/C:84 #53597e;color #2/C:85 #165ad8;color #2/C:86 #9d888d;color #2/C:87 #d04dab;color #2/C:88 #b1d5ba;color #2/C:89 #c66dab;color #2/C:90 #44a151;color #2/C:91 #df5760;color #2/C:92 #66df6f;color #2/C:93 #cf1dd8;color #2/C:94 #a269ab;color #2/C:95 #551fba;color #2/C:96 #e4a7c9;color #3/A:1 #1f5c8b;color #3/A:2 #a68a40;color #3/A:3 #5f3140;color #3/A:4 #76147c;color #3/A:5 #9d5140;color #3/A:6 #6c8ed6;color #3/A:7 #37804f;color #3/A:8 #8edb31;color #3/A:9 #7b64e5;color #3/A:10 #ab2140;color #3/A:11 #a89a40;color #3/A:12 #b53231;color #3/A:13 #7e247c;color #3/A:14 #6125d6;color #3/A:15 #a2265e;color #3/A:16 #97bd7c;color #3/A:17 #aa556d;color #3/A:18 #a9a140;color #3/A:19 #9c3931;color #3/A:20 #9e407c;color #3/A:21 #8faa6d;color #3/A:22 #166f8b;color #3/A:23 #9bd97c;color #3/A:24 #76dad6;color #3/A:25 #cd90e5;color #3/A:26 #2628d6;color #3/A:27 #97ce31;color #3/A:28 #e2c07c;color #3/A:29 #933a4f;color #3/A:30 #cf328b;color #3/A:31 #a27e5e;color #3/A:32 #b5ca31;color #3/A:33 #946222;color #3/A:34 #8f546d;color #3/A:35 #28a040;color #3/A:36 #ab4740;color #3/A:37 #8c1b6d;color #3/A:38 #95946d;color #3/A:39 #d63b6d;color #3/A:40 #76ba5e;color #3/A:41 #b5524f;color #3/A:42 #8880d6;color #3/A:43 #b3ae8b;color #3/A:44 #9acd31;color #3/A:45 #77924f;color #3/A:46 #2cc0d6;color #3/A:47 #a9a340;color #3/A:48 #e2867c;color #3/A:49 #b5b431;color #3/A:50 #498e4f;color #3/A:51 #64627c;color #3/A:52 #85188b;color #3/A:53 #e4827c;color #3/A:54 #1f1a6d;color #3/A:55 #a4a27c;color #3/A:56 #cdb2e5;color #3/A:57 #60a45e;color #3/A:58 #7d2d40;color #3/A:59 #80c45e;color #3/A:60 #444422;color #3/A:61 #7ddb40;color #3/A:62 #688240;color #3/A:63 #1fbfd6;color #3/A:64 #46c05e;color #3/A:65 #71584f;color #3/A:66 #2686d6;color #3/A:67 #353ce5;color #3/A:68 #c85b8b;color #3/A:69 #a37a31;color #3/A:70 #3518e5;color #3/A:71 #4882d6;color #3/A:72 #b55631;color #3/A:73 #74487c;color #3/A:74 #6149d6;color #3/A:75 #282c40;color #3/A:76 #211e8b;color #3/A:77 #22978b;color #3/A:78 #7998e5;color #3/A:79 #a66c40;color #3/A:80 #501931;color #3/A:81 #15dd7c;color #3/A:82 #ca756d;color #3/A:83 #3b94e5;color #3/A:84 #262cd6;color #3/A:85 #7fc322;color #3/A:86 #a88840;color #3/A:87 #2b7a8b;color #3/A:88 #486cd6;color #3/A:89 #994031;color #3/A:90 #71ce31;color #3/A:91 #3a8440;color #3/A:92 #ffffff;color #3/A:93 #a2c25e;color #3/A:94 #cf968b;color #3/A:95 #28c440;color #3/A:96 #7b3ee5;color #3/A:97 #6aa8d6;color #4/A:2 #977b82;color #4/A:3 #683191;color #4/A:4 #7dc8af;color #4/A:5 #768dcd;color #4/A:6 #bb16af;color #4/A:7 #1c5373;color #4/A:8 #3727a0;color #4/A:9 #d0afaf;color #4/A:10 #6cb755;color #4/A:11 #8d7c73;color #4/A:12 #26d755;color #4/A:13 #1f15be;color #4/A:14 #7e3464;color #4/A:15 #df35be;color #4/A:16 #484555;color #4/A:17 #9764cd;color #4/A:18 #84a191;color #4/A:19 #c766af;color #4/A:20 #6f9a55;color #4/A:21 #9eb9cd;color #4/A:22 #531582;color #4/A:23 #707064;color #4/A:24 #9b80cd;color #4/A:25 #d0dbaf;color #4/A:26 #3773a0;color #4/A:27 #26a155;color #4/A:28 #414855;color #4/A:29 #e267cd;color #4/A:30 #c023cd;color #4/A:31 #b18dbe;color #4/A:32 #84d991;color #4/A:33 #1fadbe;color #4/A:34 #bc63cd;color #4/A:35 #ad19dc;color #4/A:36 #6483cd;color #4/A:37 #c9deaf;color #4/A:38 #aab2dc;color #4/A:39 #59d182;color #4/A:40 #6d6055;color #4/A:41 #1cd955;color #4/A:42 #b5cba0;color #4/A:43 #e281af;color #4/A:44 #9537be;color #4/A:45 #d6b0be;color #4/A:46 #3bcf64;color #4/A:47 #2c6755;color #4/A:48 #e586cd;color #4/A:49 #e22dcd;color #4/A:50 #e29daf;color #4/A:51 #4935a0;color #4/A:52 #caccbe;color #4/A:53 #d573be;color #4/A:54 #e229cd;color #4/A:55 #b55782;color #4/A:56 #e449cd;color #4/A:57 #614a55;color #4/A:58 #9e69cd;color #4/A:59 #5f8873;color #4/A:60 #958fbe;color #4/A:61 #9e45cd;color #4/A:62 #975564;color #4/A:63 #1cdd73;color #4/A:64 #5ba291;color #4/A:65 #bedf55;color #4/A:66 #359564;color #4/A:67 #262d55;color #4/A:68 #711fa0;color #4/A:69 #505c64;color #4/A:70 #662773;color #4/A:71 #d319be;color #4/A:72 #e2a1cd;color #4/A:73 #b5cf82;color #4/A:74 #88b2be;color #4/A:75 #bb4aaf;color #4/A:76 #dc8773;color #4/A:77 #d579be;color #4/A:78 #7c9864;color #4/A:79 #9d6c91;color #4/A:80 #79cd64;color #4/A:81 #aa1adc;color #4/A:82 #ab4891;color #4/A:83 #1ca373;color #4/A:84 #8b4a73;color #4/A:85 #802daf;color #4/A:86 #d9c4cd;color #4/A:87 #c64daf;color #4/A:88 #857b64;color #4/A:89 #a26daf;color #4/A:90 #c629af;color #4/A:91 #adb1be;color #4/A:92 #5849af;color #4/A:93 #93b3a0;color #4/A:94 #48e155;color #4/A:95 #93d3a0;color #4/A:96 #dc4d73;color #4/A:97 #d53fbe;color #4/A:98 #6a4f55

These color commands take 15-16 seconds.

Thanks,

Mungo


by Tom Goddard, 5 years ago

Attachment: commands.txt added

comment:4 by Tom Goddard, 5 years ago

The 383 commands also fills the undo history with 383 color undo operations, not ideal. But not saving undo information reduced time only a tiny bit 2.7 seconds to 2.6 seconds.

comment:5 by Eric Pettersen, 5 years ago

Some additional timing info:

Running the color commands normally on my machine initially takes 6.5 seconds but gets slower with each repetition so that the sixth run takes 8.3 seconds, the slowdown gets worse with each try, i.e. 6.5, 6.8, 6.9, 7.3, 7.7, 8.3 . Also 25-30MB of additional memory gets used with each pass.

Running the commands with logging turned off (but not status messages) stops the progressive slowdown (i.e. all tries roughly 6.5 seconds) but the memory usage increase is still there. So it seems the repetition slowdown is from reloading a progressively bigger log. I wonder if this affects all big logs, or if its because the log is getting reloaded a zillion times here.

Running the commands with both logging and status messages turned off speeds things up to roughly a steady 4.8 seconds. The memory leak goes away. So somehow status message handling is leaking memory.

If I also have no models open when the commands are run, then the time drops to ~3.65 seconds. So actually coloring the structures is about 1.15 seconds and the rest (3.65) is overhead going from the command text to actually running the coloring code. From my previous line profiling and by-hand timings (class methods can't be line profiled), I believe it's largely from parsing the atom specs.

comment:6 by Eric Pettersen, 5 years ago

Added a regulating timer to the log to effectively collate rapid log messages. This eliminates the progressive slowing of the color commands. The memory leak and other problems are still there.

comment:7 by Eric Pettersen, 5 years ago

Repeating the color commands 5 times results in an increase in memory usage of 158 MB. With logging turned off, the same. With just the OpenGL part of the status mechanism turned off it drops to 36 MB.

in reply to:  9 comment:8 by goddard@…, 5 years ago

I will look for the status message memory leak.

comment:9 by Eric Pettersen, 5 years ago

If all I do is comment out the self._draw_text() call in _StatusBarOpenGL.status(), the memory use increase drops to 37 MB.

comment:10 by Tom Goddard, 5 years ago

The routine text_image_rgba() that uses Qt to convert status line text to an rgba image for drawing uses all the memory. It is using Qt objects QImage, QPainter, QFont, QFontMetrics, QColor, QBrush, QPen. I would expect all of these to be released when the Python objects get deleted but that is not what is happening. I will look into whether any of these need some explicit delete or release call.

comment:11 by Tom Goddard, 5 years ago

I fixed the status line memory leak. It turns out it was PyOpenGL that leaked memory when handed non-contiguous numpy arrays for textures (or for any other OpenGL call). I reported this PyOpenGL 3.1.5 bug

https://github.com/mcfletch/pyopengl/issues/47

The reason it used a non-contiguous array is the text needed to be flipped vertically to appear rightside up and that just used a vertically flipped view of the original numpy array. I protected the ChimeraX OpenGL texture and vertex buffer code making sure all numpy arrays are contiguous before calling PyOpenGL.

Still the memory use of about 15 Mbytes per execution of the 383 color commands is excessive. The log increased by 383 lines, maybe 10000 characters or 0.01 Mbytes.

comment:12 by Eric Pettersen, 5 years ago

Weird that your memory use is roughly 2X mine. Before the leak is plugged, the increase is roughly 32MB/command cycle. After being plugged it's 7MB/cycle whereas for you it's 15MB. Would like to get that near zero of course, but it's still a major improvement.

comment:13 by Tom Goddard, 5 years ago

With output to the log commented out (Log.log() -> return True in log/src/tool.py) memory use increases 6 Mbytes each time the 383 color commands are run. Here are total Mbytes used on a MacBook Pro for 9 executions of the 383 commands:

284 296 305 311 317 323 328 335 340 348

With the log not commented out it uses about 8 Mbytes per 383 color commands, so the log is adding 2 more Mbytes per run

286 301 312 316 332 339 350 352 363 361

The above timings are with the 4 models opened (open 4zho 5h92 1a70 4zhp). With no models opened, 10 tries, about 4 Mbytes per run. This suggests about 2 Mbytes per run is being used by command parsing.

223 228 238 245 246 253 251 250 252 259 262

comment:14 by Tom Goddard, 5 years ago

My memory use in previous comment is now in line with yours, about 8 Mbytes per 383 color commands with 4 models opened and log enabled. Looks like 2 Mbytes per run for logging, 2 Mbytes per run for command parsing, and 4 Mbytes per run for coloring. All of these would ideally be 0 except a tiny 0.01 for logging.

comment:15 by Eric Pettersen, 5 years ago

Replacing grako with TatSu results in an 8% speedup of atom spec parsing+evaluation (i.e. about .25 seconds aggregate total). Not breathtaking. Still, why not? Will be tougher to evaluate parsimonious, since it uses a different grammar and different parse-tree handling.

comment:16 by Eric Pettersen, 5 years ago

On the other hand, TatSu does not correctly parse our command language, so scratch that (it doesn't correctly parse commands with an empty/omitted argument, e.g. "hb #1 two true" is okay but not "hb two true").

comment:17 by Eric Pettersen, 5 years ago

Looked at _why_ TatSu doesn't work a little bit. It's because the parser reports the failure position as after the first token rather than before it, so the 'Or' argument parser is fooled into thinking that the atom spec was partially parsed and that it therefore _is_ an atom spec, just mangled, rather than not an atom spec at all. So it never goes on to try the second option for the 'Or', it just reports an error.

comment:18 by Eric Pettersen, 3 years ago

Seems like we are going to have to use a C++ -based parser. I translated the atomspec syntax rules for Lark, supposedly one of the fastest pure Python parsers, and it takes 3.8 seconds to generate the parse tree. I've attached the test file.

by Eric Pettersen, 3 years ago

Attachment: atomspec.lalr added

comment:19 by Eric Pettersen, 3 years ago

Reporter: changed from g.m.carstairs@… to Eric Pettersen

Mungo has retired

comment:20 by Tom Goddard, 3 years ago

Is the 3.8 seconds with Lark for the long color command in comment 3?

comment:21 by Eric Pettersen, 3 years ago

Yes, it's for the long series of color commands in comment 3 (you can see it in the attachment).

comment:22 by Eric Pettersen, 3 years ago

My comment about "going to have to use a C++-based parser" was slightly off base. There is no way on God's green earth that we're going to move the entirety of command-line parsing into C++. That would be nigh impossible with aliasing, Annotation inheritance, etc. Moving just the atom-spec parsing into C++ is still worth doing because of long atom specs crashing ChimeraX (#4842) and that may speed up command parsing some, but we may have to live with relative slow command parsing. I guess do #4842 first and see how much that helps.

Note: See TracTickets for help on using tickets.