Ticket #20831: Boltz_error

File Boltz_error, 10.3 KB (added by swlovell@…, 4 hours ago)

Added by email2trac

Line 
1Running Boltz prediction of protein with 275 residues, 1 ligands C[C@H](CCC(O)=O)[C@H]1CC[C@H]2[C@@H]3CC[C@@H]4C[C@H](O)CC[C@]4(C)[C@H]3CC[C@]12C on gpu
2Using cached multiple sequence alignment /home/swlovell/Downloads/ChimeraX/BoltzMSA/TANGO2_lithocholic
3Running boltz prediction failed with exit code 1:
4command:
5/home/swlovell/boltz22/bin/boltz predict /home/swlovell/structures/Gonzalez_Lina/TANGO2/Docking/boltz2/Lithocholic_acid/TANGO2_lithocholic_acid.yaml --use_potentials --accelerator gpu
6stdout:
7Boltz version 2.2.0
8Checking input data.
9All inputs are already processed.
10Processing 0 inputs with 0 threads.
11Running structure prediction for 1 input.
12Predicting: | | 0/? [00:00<?, ?it/s] Predicting: 0%| | 0/1 [00:00<?, ?it/s] Predicting DataLoader 0: 0%| | 0/1 [00:00<?, ?it/s] Predicting DataLoader 0: 0%| | 0/1 [00:01<?, ?it/s]
13stderr:
140it [00:00, ?it/s] 0it [00:00, ?it/s]
15Using bfloat16 Automatic Mixed Precision (AMP)
16GPU available: True (cuda), used: True
17TPU available: False, using: 0 TPU cores
18HPU available: False, using: 0 HPUs
19/home/swlovell/boltz22/lib/python3.11/site-packages/pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py:76: Starting from v1.9.0, `tensorboardX` has been removed as a dependency of the `pytorch_lightning` package, due to potential conflicts with other packages in the ML ecosystem. For this reason, `logger=True` will use `CSVLogger` as the default logger, unless the `tensorboard` or `tensorboardX` packages are found. Please `pip install lightning[extra]` or one of them to enable TensorBoard support by default
20Fri Aug 7 05:37:27 2026: Loading Boltz structure prediction weights
21/home/swlovell/boltz22/lib/python3.11/site-packages/pytorch_lightning/utilities/migration/utils.py:56: The loaded checkpoint was produced with Lightning v2.5.0.post0, which is newer than your current Lightning version: v2.5.0
22Fri Aug 7 05:37:38 2026: Finished loading Boltz structure prediction weights
23Fri Aug 7 05:37:38 2026: Starting structure inference
24You are using a CUDA device ('NVIDIA RTX PRO 4500 Blackwell') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
25LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
26Fri Aug 7 05:37:39 2026: Begin structure inference TANGO2_lithocholic_acid (1 of 1)
27Traceback (most recent call last):
28File "/home/swlovell/boltz22/bin/boltz", line 8, in <module>
29sys.exit(cli())
30^^^^^
31File "/home/swlovell/boltz22/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
32return self.main(*args, **kwargs)
33^^^^^^^^^^^^^^^^^^^^^^^^^^
34File "/home/swlovell/boltz22/lib/python3.11/site-packages/click/core.py", line 1078, in main
35rv = self.invoke(ctx)
36^^^^^^^^^^^^^^^^
37File "/home/swlovell/boltz22/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
38return _process_result(sub_ctx.command.invoke(sub_ctx))
39^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
40File "/home/swlovell/boltz22/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
41return ctx.invoke(self.callback, **ctx.params)
42^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
43File "/home/swlovell/boltz22/lib/python3.11/site-packages/click/core.py", line 783, in invoke
44return __callback(*args, **kwargs)
45^^^^^^^^^^^^^^^^^^^^^^^^^^^
46File "/home/swlovell/boltz22/lib/python3.11/site-packages/boltz/main.py", line 1355, in predict
47trainer.predict(
48File "/home/swlovell/boltz22/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 859, in predict
49return call._call_and_handle_interrupt(
50^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
51File "/home/swlovell/boltz22/lib/python3.11/site-packages/pytorch_lightning/trainer/call.py", line 47, in _call_and_handle_interrupt
52return trainer_fn(*args, **kwargs)
53^^^^^^^^^^^^^^^^^^^^^^^^^^^
54File "/home/swlovell/boltz22/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 898, in _predict_impl
55results = self._run(model, ckpt_path=ckpt_path)
56^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
57File "/home/swlovell/boltz22/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 982, in _run
58results = self._run_stage()
59^^^^^^^^^^^^^^^^^
60File "/home/swlovell/boltz22/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 1021, in _run_stage
61return self.predict_loop.run()
62^^^^^^^^^^^^^^^^^^^^^^^
63File "/home/swlovell/boltz22/lib/python3.11/site-packages/pytorch_lightning/loops/utilities.py", line 179, in _decorator
64return loop_run(self, *args, **kwargs)
65^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
66File "/home/swlovell/boltz22/lib/python3.11/site-packages/pytorch_lightning/loops/prediction_loop.py", line 125, in run
67self._predict_step(batch, batch_idx, dataloader_idx, dataloader_iter)
68File "/home/swlovell/boltz22/lib/python3.11/site-packages/pytorch_lightning/loops/prediction_loop.py", line 255, in _predict_step
69predictions = call._call_strategy_hook(trainer, "predict_step", *step_args)
70^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
71File "/home/swlovell/boltz22/lib/python3.11/site-packages/pytorch_lightning/trainer/call.py", line 323, in _call_strategy_hook
72output = fn(*args, **kwargs)
73^^^^^^^^^^^^^^^^^^^
74File "/home/swlovell/boltz22/lib/python3.11/site-packages/pytorch_lightning/strategies/strategy.py", line 438, in predict_step
75return self.lightning_module.predict_step(*args, **kwargs)
76^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
77File "/home/swlovell/boltz22/lib/python3.11/site-packages/boltz/model/models/boltz2.py", line 1059, in predict_step
78out = self(
79^^^^^
80File "/home/swlovell/boltz22/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
81return self._call_impl(*args, **kwargs)
82^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
83File "/home/swlovell/boltz22/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
84return forward_call(*args, **kwargs)
85^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
86File "/home/swlovell/boltz22/lib/python3.11/site-packages/boltz/model/models/boltz2.py", line 464, in forward
87z = z + template_module(
88^^^^^^^^^^^^^^^^
89File "/home/swlovell/boltz22/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
90return self._call_impl(*args, **kwargs)
91^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
92File "/home/swlovell/boltz22/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
93return forward_call(*args, **kwargs)
94^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
95File "/home/swlovell/boltz22/lib/python3.11/site-packages/boltz/model/modules/trunkv2.py", line 498, in forward
96v = v + self.pairformer(v, pair_mask, use_kernels=use_kernels)
97^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
98File "/home/swlovell/boltz22/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
99return self._call_impl(*args, **kwargs)
100^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
101File "/home/swlovell/boltz22/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
102return forward_call(*args, **kwargs)
103^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
104File "/home/swlovell/boltz22/lib/python3.11/site-packages/boltz/model/layers/pairformer.py", line 329, in forward
105z = layer(
106^^^^^^
107File "/home/swlovell/boltz22/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
108return self._call_impl(*args, **kwargs)
109^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
110File "/home/swlovell/boltz22/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
111return forward_call(*args, **kwargs)
112^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
113File "/home/swlovell/boltz22/lib/python3.11/site-packages/boltz/model/layers/pairformer.py", line 243, in forward
114z = z + dropout * self.tri_mul_out(
115^^^^^^^^^^^^^^^^^
116File "/home/swlovell/boltz22/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
117return self._call_impl(*args, **kwargs)
118^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
119File "/home/swlovell/boltz22/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
120return forward_call(*args, **kwargs)
121^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
122File "/home/swlovell/boltz22/lib/python3.11/site-packages/boltz/model/layers/triangular_mult.py", line 92, in forward
123return kernel_triangular_mult(
124^^^^^^^^^^^^^^^^^^^^^^^
125File "/home/swlovell/boltz22/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
126return fn(*args, **kwargs)
127^^^^^^^^^^^^^^^^^^^
128File "/home/swlovell/boltz22/lib/python3.11/site-packages/boltz/model/layers/triangular_mult.py", line 23, in kernel_triangular_mult
129return triangle_multiplicative_update(
130^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
131File "/home/swlovell/boltz22/lib/python3.11/site-packages/cuequivariance_torch/primitives/triangle.py", line 314, in triangle_multiplicative_update
132return f(
133^^
134File "/home/swlovell/boltz22/lib/python3.11/site-packages/cuequivariance_ops_torch/__init__.py", line 97, in triangle_multiplicative_update
135_raise_triton_import_error("triangle_multiplicative_update")
136File "/home/swlovell/boltz22/lib/python3.11/site-packages/cuequivariance_ops_torch/__init__.py", line 91, in _raise_triton_import_error
137raise Exception(
138Exception: Failed to import Triton-based component: triangle_multiplicative_update:
139Traceback (most recent call last):
140File "/home/swlovell/boltz22/lib/python3.11/site-packages/cuequivariance_ops_torch/attention_pair_bias.py", line 27, in <module>
141from torch.fx._symbolic_trace import is_fx_symbolic_tracing
142ImportError: cannot import name 'is_fx_symbolic_tracing' from 'torch.fx._symbolic_trace' (/home/swlovell/boltz22/lib/python3.11/site-packages/torch/fx/_symbolic_trace.py)
143
144During handling of the above exception, another exception occurred:
145
146Traceback (most recent call last):
147File "/home/swlovell/boltz22/lib/python3.11/site-packages/cuequivariance_ops_torch/__init__.py", line 64, in <module>
148from cuequivariance_ops_torch.attention_pair_bias import (
149File "/home/swlovell/boltz22/lib/python3.11/site-packages/cuequivariance_ops_torch/attention_pair_bias.py", line 29, in <module>
150from torch.fx._symbolic_trace import (
151ImportError: cannot import name 'is_fx_tracing_symbolic_tracing' from 'torch.fx._symbolic_trace' (/home/swlovell/boltz22/lib/python3.11/site-packages/torch/fx/_symbolic_trace.py)
152
153Please make sure to install triton==3.3.0. Other versions may not work!
154sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
155