Make Acer SpatialLabs View Pro 27" 3D display work with OpenXR

[Tom Goddard]

Krish Raman of Biocryst has glasses-free Acer SpatialLabs View Pro 27" 3d display. It is not working with ChimeraX OpenXR (command "xr on"), the device seems to disconnect after a successful OpenXR connection is made. I've put code in to give a clear error message when the device disconnects. I would need to have a display on my desk to debug this further.

Rex Lin of Acer has offered to deliver a display to UCSF from their San Jose location so I can try to fix the OpenXR problems. His boss Jane Hsu may deliver the display on March 10 or 11, 2025.

ChimeraX OpenXR works with a similar Sony Spatial Reality 3D display so it may be relatively easy to get it working with the Acer display.

[Tom Goddard]

Ticket #16864 describes improvements to ChimeraX OpenXR needed to make it work better with the Sony Spatial Reality display. Those will also be needed to work well with the Acer display.

[Tom Goddard]

Jane Hsu delivered an Acer 3d display today and demonstrated ChimeraX working with their SteamVR bridge.

I tested it with ChimeraX OpenXR and find that xrLocateViews() always returns 0 views instead of the 2 expected stereoscopic views. That call also says the position and orientation are valid. I tested pyopenxr_examples hello_xr and it too gives an error saying xrLocateViews() returned 0 views. I've contacted SpatialLabs developer support and made a ticket asking for help with this

​https://developer-spatiallabs.atlassian.net/servicedesk/customer/portal/1/DS-179

I'll attach the hello_xr output to this ticket.

[Tom Goddard]

Output from pyopenxr_examples github program hello_xr showing xrLocateViews() problem.

[Tom Goddard]

Screenshot of my ticket to SpatialLabs support.

[Tom Goddard]

Acer tech support sent OpenXR/OpenGL example C++ code. Might be able to see how that code differs from the ChimeraX code.

I have a new theory about what is wrong with xrLocateViews(). In the C API it takes a view count and array of views argument. But in the pyopenxr API it eliminates those two arguments and presumably figures out that there should be two views for a stereo rendering mode (which is another xrLocateViews argument). My guess is that it may be figuring out that it expects 2 views using another OpenXR call that Acer OpenXR did not implement and the Acer OpenXR always returns 0. It would be worth compiling my own pyopenxr if possible to print out what number of views it is passing to the xrLocateViews() C call. This explanation would account for why we see the problem in the pyopenxr hello_xr example and ChimeraX but other apps like Blender or the Acer C++ sample code do not hit it because they hardcode that 2 views are expected.

[Tom Goddard]

The first Acer OpenXR is as surmised in the previous comment. PyOpenXR is querying the number of views by calling xrLocateViews() with view_capacity_input 0 and Acer OpenXR incorrectly returns 0 instead of 2. This should be reported to Acer tech support so they can fix it. In the meantime I use a custom version of PyOpenXR xr.locate_views() that hardcodes the number of views to 2.

After this fix I get molecules rendering to the Acer display but only for a few seconds before I get an opengl error on glClear() about incomplete framebuffer. This appears to be because the swapchain texture provided by Acer OpenXR is invalid. The shell where ChimeraX started gives clues issuing errors:

glImportMemoryWin32HandleEXT: Fail. Error Code: 0
glTextureStorageMem2DEXT: Fail. Error Code: 0

Seems to be an out of memory condition. So I printed the texture id after acquire_swapchain_image() and wait_swapchain_image() are called on each eye rendering. Unexpectedly I see texture ids increasing on every render 312 times with tex ids from 14 to 943 before the above memory errors occur and then the glClear failure probably because the texture allocation failed.

When I run the same with the Sony display it simply alternates between two texture ids for hundreds of frames. And a comment suggests that Meta OpenXR cycles between 3 texture ids.

So the question is why does Acer OpenXR appear to allocate an ever increasing number of textures? At the end of each render the swapchain image is released and maybe Acer OpenXR actually then won't reuse it. I should check whether the Acer OpenXR example code releases the swapchain images.

More debug output shows the swapchain image count is always 3 and the swapchain index is always 0,1, or 2. But the texture ids keep rising on every render. I use a separate swapchain for each eye. Maybe that is the issue, switching between swapchains may be where Acer OpenXR goes haywire. Maybe it is only implemented to work efficiently with one swapchain. I changed the code to use just one swapchain and it still increases the texture ids on every render and wipes out in the same way with the memory errors.

The Acer example OpenXR code uses a single swapchain. There is some alarming code in my _create_swapchains() routine that adds the swapchain images to a list with a comment saying it is to keep the images alive. That list is never used by saved as an attribute. But it seems as if a deallocation was happening in Python that led to the textures being recreated it would happen on the Sony display too. It would be useful to know if the Acer example code keeps creating new textures. That would be some work to compile the C++ on Windows.

I see a possible critical difference with the Acer sample code. It calls xrEnumerateSwapchainImages() only at initialization while my code calls it every frame. Test if calling it only at initialization fixes the problem. That does fix it. Now ChimeraX renders correctly on the display using "Screen Mode" in Acer settings (Alt+C when in stereo).

In "VR Mode" from Acer settings it kind of works but the depth is compressed depending on how far out of the screen I move the model. It can look correct in VR Mode moved far in front of screen but looks flattened in depth if I try to move the model to be half in front and half in back of screen. This probably can probably be figured out with reference to the Acer explanation of the camera views used in these two views and probably it can be fixed to work in either mode, although screen mode may be the preferred mode for ChimeraX viewing.

[Tom Goddard]

Summarizing there are two bugs in the Acer OpenXR runtime that prevented ChimeraX from rendering on the display. xrLocateViews() reported the wrong number of stereo views (0 instead of 2). And xrEnumerateSwapchainImages() keeps recreating new textures on every call instead returning the already allocated textures. I'll report both of these problems on the Acer OpenXR support ticket and hopefully they can fix them. I can work around these problems in the ChimeraX OpenXR code if necessary, using the work-arounds only in the case an Acer display is detected.

[Tom Goddard]

From: "Hsu, Jane"
Subject: RE: [chimerax-users] Regarding Chimerax and Acer SpatialLabs 3D display/Acer OpenXR runtime
Date: April 27, 2025 at 8:23:42 PM PDT
To: Tom Goddard <goddard@…>
Cc: "Wang, Yuli", "Lin, Rexx"

Hi Tom,

Apologies for the delayed reply. I hope you had a great trip to Guizhou and that the earthquake didn’t affect you.
Thank you for the ticket — we’ve received it and passed it on to our internal team. The OpenXR team is currently fully occupied with technical collaborations with silicon vendors. I’m pushing to get some of their resources allocated to look into these bugs.

Would you mind holding on to our device a bit longer? We’ll keep you posted once the bugs are fixed. It looks like it’s going to take longer than I originally expected. Many thanks for your patience.

Cheers,
Jane

[Tom Goddard]

Acer has provided a new OpenXR runtime with fixes for the bugs I reported. I will test in early July when I have access to the Acer display if this fixes the problems. If they are fixed I will try to add support for the Acer display in the ChimeraX daily build.

Begin forwarded message:

From: Developer Support <jira@…>
Subject: Developer support DS-179 OpenXR xrLocateViews() returns no views with Acer SpatialLabs View Pro 27 display
Date: June 19, 2025 at 5:57:10 AM EDT
To: "goddard@…" <goddard@…>
Reply-To: support@…

Developer Support commented:

Hi Tom

We prepared a test build of Acer OpenXR Runtime to support PyOpenXR for ChimeraX.
This runtime has been verified with ChimeraX1.9 to visualize moleculars in stereoscopic 3d.
Kindly update AcerXR by following the steps outlined below:

Manually execute Uninstall.exe from the path: *C:\Program Files\Acer\Acer XR Service*
Install the attached file “AcerXRService.1.01.2059.installer.msi“ to update the runtime, then reboot your system.
AcerXRService.1.01.2059.installer.msijira-generated-image-static-link_attachment_7-606c5ac5-df7c-47a8-a00d-5d9f98f082bd.gif
Launch ChimeraX to open a structure, then execute “xr on“ in the command line.
By default it should display stereoscopic 3d as VR Mode.
You could press Alt+C to open Acer OpenXR Control Panel for detail settings of OpenXR.
​https://spatiallabs.acer.com/developer/docs/2299cdda-f90f-11ed-b3b8-067bb43818a8/47b12225-07f8-48f5-8a8d-cfdf99700259
We recommend choosing Screen Mode to maintain the focus plane at the screen's position.
Modify Zoom parameter to ensure the model size remains within the screen's dimensions.
The command line "xr off" or use "Alt+T" to exit stereoscopic 3d.
jira-generated-image-static-883e82c6-9d38-4c8d-bf27-f9492518413b.png

The 3D effect is reaaly impressive, and we expect SpatialLabs display could broaden possibilities of ChimeraX's application.

BR,
SpatialLabs Team

SpatialLabs Team
View request · Turn off this request's notifications

[Tom Goddard]

I tested the Acer OpenXR runtime installed with AcerXRService.1.01.2059.installer.msi and the two openxr bugs mentioned in comment 5 are fixed.

[Tom Goddard]

I added ChimeraX OpenXR code to center models on Acer display.

[Tom Goddard]

I tried to get mouse events in the full-screen window in the Acer 3d display to allow rotation, translation and zooming when the mouse is moved to that display. Did not succeed.

I could find the Acer 3d window handle by enumerating all top level windows and taking the new one that appears after I use "xr on":

from win32 import win32gui

def callback(hwnd, extra):
    # Get the window title
    title = win32gui.GetWindowText(hwnd)
    # Check if the window is visible and has a title
    if win32gui.IsWindowVisible(hwnd) and title:
        print(f"Window Handle: {hwnd}, Title: {title}")
    return True # Continue enumeration
    
win32gui.EnumWindows(callback, None)

The new one had title "FloatingWindow":

Window Handle: 198444, Title: FloatingWindow
Window Handle: 198038, Title: ChimeraX Python Shell
Window Handle: 132376, Title: ChimeraX
Window Handle: 132164, Title: pywin32 find toplevel windows - Google Search — Mozilla Firefox
Window Handle: 67102, Title: Settings
Window Handle: 67080, Title: Settings
Window Handle: 66402, Title: *shell* - GNU Emacs at vizvault
Window Handle: 262834, Title: Windows Input Experience
Window Handle: 65848, Title: Program Manager

I could then create a QWindow using

from Qt.QtGui import QWindow

w = QWindow.fromWinId(198444)

w.height()
2160
w.width()
3840

w.windowStates()
<WindowState.WindowNoState: 0>
w.title()
''
w.flags()
<WindowType.ForeignWindow: 33>

s = w.screen()
s.name()
'ASV27-2P'
s.manufacturer()
'Acer Technologies'
s.model()
'ASV27-2P'

I was not able to get any mouse events, wheel events, focus in/out events, or any events at all from this QWindow. The following did not allow zoom.

w.wheelEvent = session.ui.mouse_modes._wheel_event

Trying to print events also printed nothing when mousing in the Acer 3D full screen window

def we(event):
    print('wheel event', event)
    
w.wheelEvent = we
w.event = we
w.mousePressEvent = we

Creating a widget container for the window did not allow getting any events. After adding a QWidget the window gets a frame and gets the focus when clicked on. But this new top level window is not shown by Windows task manager. Maybe the Acer OpenXR created this window with some special flags which prevent it from receiving mouse input.

from Qt.QtWidgets import QWidget
c = QWidget.createWindowContainer(w)
c.show()
c.wheelEvent = we

The QWindow.fromWinId() documentation says

Note: The resulting QWindow should not be used to manipulate the underlying native window
(besides re-parenting), or to observe state changes of the native window. Any support for
these kind of operations is incidental, highly platform dependent and untested.

Maybe Qt does not register the needed handlers to get window events when creating the QWindow.

It may be possible to use the native Windows API to get mouse events using MouseProc and SetWindowsHookEx. This seems like a lot of work to try to get mouse events.

[Tom Goddard]

Google search (pywin32 mouseproc example) gave the following AI generated example code for installing a low level mouse handler with pywin32. I did not try it.

import win32api
import win32con
import win32gui
import win32hooks

# Define your callback function
def LowLevelMouseProc(nCode, wParam, lParam):
    if nCode >= 0:
        # Process the mouse event here
        msg = wParam # Mouse message (e.g., WM_LBUTTONDOWN)
        hook_struct = lParam # Pointer to MSLLHOOKSTRUCT
        print(f"Mouse event: {msg}, Position: ({hook_struct.pt.x}, {hook_struct.pt.y})")

    # Always call CallNextHookEx
    return win32hooks.CallNextHookEx(None, nCode, wParam, lParam)

# Install the hook
hook_id = win32hooks.SetWindowsHookEx(
    win32con.WH_MOUSE_LL,  # Hook type: Low-level mouse events
    LowLevelMouseProc,     # Your callback function
    win32api.GetModuleHandle(None), # Handle to the DLL containing the hook procedure
    0                      # 0 to monitor all threads on the desktop
)

# Run a message loop (required for the hook to work)
msg = win32gui.GetMessage(None, 0, 0)
while msg[0]:
    win32gui.TranslateMessage(msg[0])
    win32gui.DispatchMessage(msg[0])
    msg = win32gui.GetMessage(None, 0, 0)

# Unhook when done
win32hooks.UnhookWindowsHookEx(hook_id)

[Tom Goddard]

It might be worth trying the Sony 3D display to see if it will provide mouse events through a QWindow. This could work if the Acer OpenXR is creating the window in such a way that it does not get mouse events while the Sony OpenXR creates its window and allows mouse events. I think more likely it is Qt that is not registering for mouse events since it did not create the window.

[Tom Goddard]

Another option that could be the most feasible is to create a new Qt window fullscreen on the Acer 3d display that sits behind the Acer OpenXR window. I think it will get the mouse events. When I deleted the QWidget I created for the OpenXR window it repositioned the QWindow over my 2D display covering the ChimeraX GUI. But I could still rotated the molecule if I put the mouse over the OpenXR window in the region where the underlying ChimeraX graphics pane was. I would want the backing window to be full-screen without a titlebar, and I would need to destroy it when OpenXR turned off.

I tested this and it works. The Acer OpenXR window is always on top so no need to lower the backing window.

s = session.ui.screens()[1]  # Should choose by name
w = QWidget()
w.move(s.geometry().topLeft())
w.showFullScreen()
w.wheelEvent = lambda e: print('wheel event', e)

The next problem is that the ChimeraX MouseMode instances uses the main view for both determining the window size and for adjusting the camera. So the mouse binding on the Acer screen should probably first convert the mouse event x,y to the equivalent x,y on the main graphics window and then call the main graphics window mouse event dispatching. Since the main graphics window is mirroring one of the Acer eye views this should allow everything (rotate, translate, zoom, select, pick, ...) to work. One glitch will be that the z_rotation uses the main graphics window bounds and if the aspect ratios of the main graphics and the acer screen differ (which they will) then the z-rotation will have wider zones on the Acer display either at the left/right (if Acer display has wider aspect) or top/bottom (if Acer has taller aspect). That seems ok.

[Tom Goddard]

Python script to create a backing window on Acer 3d display that handles mouse events.

[Tom Goddard]

I tested Python code (attached as mouse.py) that creates a backing window on the Acer display that handles mouse events when the mouse is on the Acer display. This is pretty nice. Even selection works (aligning with the right eye).

Next step might be to incorporate this into the "xr" command. Also test if the same code works for the Sony 3d display.

Might also want keypress events forwarded to command line so commands can be typed and up/down arrow can be used on selections.

[Tom Goddard]

I made the "xr on" command enable mouse and keyboard input on the Acer and Sony 3D displays.

To select or pick an atom the mouse pointer needs to align with the object using the right eye. This is because the mouse events are mapped to the main graphics pane and it mirrors the right eye view from the 3D screen.

This is done on the Acer display using a backing Qt window which captures the mouse and key events since the Acer OpenXR window passes those events through to the window behind. The Sony SpatialReality display OpenXR window captures the mouse and key events so on that display I make a transparent Qt windows that lies in front of the Sony OpenXR window to capture mouse and key events.

[Tom Goddard]

One further improvement for the Acer and Sony 3D displays that I would like to make is to have mouse hover show the atom or residue under the mouse on the 3D display as it does on the normal 2D graphics panel. I tried this but the mouse hover code specifically looks for mouse cursor positions (QCursor.pos()) over the main graphics pane so it does not detect hover on the 3D displays. It will require some changes to the hover code to allow it to work with the 3D displays.

I tried making the hover code place popup labels on the Acer display and was unable to get them to appear on top of the graphics. I sometimes see them flash briefly. Also dragging the ChimeraX window to the Acer display puts it below the graphics and invisible. And tool tips from the Toolbar don't appear either. I tried the Qt.WindowStaysOnTop flag and Qt.WA_AlwaysStackOnTop attribute on the popup Tooltip that shows atom specs but neither helped.

Probably the tooltip popup would appear on the Sony display. But I think the appearance will be unpleasant because it does not have the correct depth. So I experimented using the Label right mouse mode. This shows the labels with correct depth and looked good. And it had the advantage that the labels remain when rotated and are cleared manually by clicking on the background. The one drawback was that the white text on black background was a bit hard to read. Making a black label background improved legibility a lot. It would be nice to add a mouse mode option for label to set the background color. Might be desirable to set the right mouse mode to label with black background when openxr display is enabled.

[Tom Goddard]

I made mouse hover on the 3D Acer or Sony screen show an atom, residue or bond label. This is an actual 3d label rather than a popup tool tip window, so it appears at the correct depth on the 3d screen. I made the label have a black background since that improves visibility.