job folder

/var/tmp/cxwebservices/{job_id}

status

​https://webservices.rbvi.ucsf.edu/cxservices/api/v1/chimerax/jobs/{job_id}/status

list of files

​https://webservices.rbvi.ucsf.edu/cxservices/api/v1/chimerax/jobs/{job_id}/files

specific file

​https://webservices.rbvi.ucsf.edu/cxservices/api/v1/chimerax/jobs/{job_id}/files/{file_name}

log files

/usr/local/www/logs/webservices-httpd/{,/cxservices}/*log

workers

pcs status, ssh to that host, run 'rq info'

rq info output will look something like

default      | 0, 0 executing, 126 finished, 7053 failed
1 queues, 0 jobs total

eaddc063617d461b85e534e32666f21c (crick.cgl.ucsf.edu 127.0.0.1:45442 865640): idle default. jobs: 33 finished, 0 failed
e85f2a71082847d2b4f3086c5d969638 (crick.cgl.ucsf.edu 127.0.0.1:45482 865637): idle default. jobs: 26 finished, 1 failed
efbda960ca184972843b53033eb2499f (crick.cgl.ucsf.edu 127.0.0.1:45630 865631): idle default. jobs: 18 finished, 1 failed
778d02349f354213ae1845532bdb3e95 (crick.cgl.ucsf.edu 127.0.0.1:45562 865624): idle default. jobs: 30 finished, 1 failed
d9db63667bc24a5ea3eb47b7d427ab58 (crick.cgl.ucsf.edu 127.0.0.1:45528 865638): idle default. jobs: 29 finished, 6 failed
8ae748e7c1b14d7195ce32bc11252c9f (crick.cgl.ucsf.edu 127.0.0.1:45594 865634): idle default. jobs: 24 finished, 5 failed
2e9469079f714a30a25b5d10e2bfc5d7 (crick.cgl.ucsf.edu 127.0.0.1:45658 865629): idle default. jobs: 24 finished, 2 failed
83fec5cc8dfb48b5af3fa49a81481472 (crick.cgl.ucsf.edu 127.0.0.1:45458 865633): idle default. jobs: 27 finished, 6 failed
4c34881336d643ad972337c5a22ef74f (crick.cgl.ucsf.edu 127.0.0.1:45662 865645): idle default. jobs: 23 finished, 1 failed
dbebb555913242ec912112eb4263f331 (crick.cgl.ucsf.edu 127.0.0.1:45556 865642): idle default. jobs: 27 finished, 3 failed
d58bd306665d464b8181af3f522f4fd9 (crick.cgl.ucsf.edu 127.0.0.1:45468 865643): idle default. jobs: 24 finished, 3 failed
135c2cbe686a4a2abc6aec16ab6024f0 (crick.cgl.ucsf.edu 127.0.0.1:45512 865641): idle default. jobs: 23 finished, 1 failed
e2e606b52b784a87b627535662ec615d (crick.cgl.ucsf.edu 127.0.0.1:45608 865626): idle default. jobs: 30 finished, 4 failed
85a3395ad77943158b45eaac50dc5dcd (crick.cgl.ucsf.edu 127.0.0.1:45628 865636): idle default. jobs: 25 finished, 2 failed
1f496e12fb1548e5b3a39f5df5bbd5f8 (crick.cgl.ucsf.edu 127.0.0.1:45524 865639): idle default. jobs: 23 finished, 2 failed
5f343863c41c4eedb1115d30b6296861 (crick.cgl.ucsf.edu 127.0.0.1:45538 865632): idle default. jobs: 29 finished, 3 failed
6c30564cda4741f59e4b7eae39198a92 (crick.cgl.ucsf.edu 127.0.0.1:45620 865627): idle default. jobs: 24 finished, 1 failed
c12e829c4ddf450db4fffe6640747047 (crick.cgl.ucsf.edu 127.0.0.1:45496 865644): idle default. jobs: 32 finished, 2 failed
9e2d6b3b2b3147b6aa9132dd2b197fca (crick.cgl.ucsf.edu 127.0.0.1:45426 865635): idle default. jobs: 33 finished, 1 failed
95cc8b75f0ec4b648523d7d05761d22b (crick.cgl.ucsf.edu 127.0.0.1:45582 865630): idle default. jobs: 29 finished, 3 failed
20 workers, 1 queues

Updated: 2026-01-09 00:47:36.351286

Usually workers are either all dead or all alive. They will all die if /usr/local/www/logs is not writable by Apache.

If you want to check what's causing job failures look at /usr/local/www/logs/webservices-httpd/cxservices/rq-worker-{id}.log