<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi Ben,<div class=""><br class=""></div><div class=""> One more bit of useful info regarding rendering large maps with solid style. The “perpendicular to view” projection mode is not the same as the mode “x, y, or z planes”. The perpendicular mode uses an OpenGL graphics 3d texture and slices the map perpendicular to the viewing direction and uses trilinear interpolation on the graphics card. The x,y,z planes mode uses OpenGL 2d textures perpendicular to one of the x, y, or z axes. While the perpendicular mode seems better the trouble is that 3d textures are generally not used in video games, so this is a very obscure graphics feature that may not be well optimized in the graphics driver, it may be slow (e.g. be done with no hardware acceleration), it may have limits to the size of the volume. On the other hand 2d textures are used in every video game and are optimized to the hilt — they are what is used to put color patterns on any surface. With some graphics cards and drivers the 3d textures will work as well as 2d textures, particularly the high-end cards, but you will only find this out by trial and error. The default “auto” projection mode never uses perpendicular to view because it is a bad choice on many low-end graphics cards.</div><div class=""><br class=""></div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>Tom</div><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Dec 18, 2015, at 1:38 AM, Benoît Zuber <<a href="mailto:benoit.zuber@ana.unibe.ch" class="">benoit.zuber@ana.unibe.ch</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type" class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
Hi Tom<br class="">
<br class="">
Thanks for the additional info. I didn't know about the projection
mode option. I was wondering before why under some conditions z
planes were super thin when viewed perpendicular to z axis and why
sometimes the gaps were filled. This was a bit problematic for some
data sets where we actually mainly look in this orientation. Now
using projection mode "perpendicular to view" instead of auto forces
the planes to be rendered thick. great. <br class="">
<br class="">
watching nvidia-smi outputs confirms that only one GPU is used for
rendering. Watching is also very useful to know how much the volume
must be cropped before switching to the next step size without
saturating the GPU memory. <br class="">
Now I got it, if we want to watch bigger volumes, we just need to
get a top geforce with 12 Gb instead of 3 Gb (Santa if you can hear
me)<br class="">
<br class="">
Cheers<br class="">
Ben<br class="">
<div class="moz-cite-prefix">On 18.12.2015 01:02, Tom Goddard wrote:<br class="">
</div>
<blockquote cite="mid:EDD5A94B-9635-453C-A3EC-B797EC22CDB1@sonic.net" type="cite" class="">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" class="">
Hi Ben,
<div class=""><br class="">
</div>
<div class=""> From what I read both Nvidia SLI and AMD CrossFire
for using multiple graphics cards both require hardware jumpers
between the cards.</div>
<div class=""><br class="">
</div>
<div class=""> The Chimera solid benchmark makes a cube shaped
volume and times rendering it. A 1024 cubed volume takes 2
Gbytes on the graphics card because each grid point takes two
bytes, one for the gray level and one for the transparency. An
important thing to know about the solid rendering with really
big volumes is that by default when you rotate it will make up
to 3 full copies of the data — so up to 6 Gbytes in the
benchmark example. It makes a copy when you rotate to viewing a
different face of the cube. It then treats it as slices
parallel to that face. So for a cube shaped map you will see a
freeze when you rotate beyond 45 degrees. For non-cube maps I
think the rule is it uses the face of the box that is showing
the most area. For big volumes if you don’t care so much about
looking at it along x and y directions then you might change
projection mode the Features / Solid Rendering Options from
“auto” to “z planes”. You probably want to know what auto does
— it uses z planes if the volume is a thin slab, but "x, y, or z
planes” if it is not a thin slab. Thin means x or y at least 4
times larger than z (I pressed Help on Volume Viewer dialog to
learn that bit, even though I wrote the code). Once “projection
mode” is “z plane” then you won’t get any of the extra copies.</div>
<div class=""><br class="">
</div>
<div class=""> For really big volumes I could probably the
transparency and use one byte per grid point. I could just
scale transparency with intensity (which is what volume viewer
does anyways). That makes the data of the graphics card half as
big so it would be possible to show larger maps. I’ll try that
in our next generation ChimeraX software — would be a nice
improvement.</div>
<div class=""><br class="">
</div>
<div class=""> Back to using multiple graphics cards — I don’t
think the driver can possibly split the volume solid rendering
data (2d OpenGL textures) — it will try to put all of it on both
graphics cards — so you won’t get more memory by using more
cards.</div>
<div class=""><br class="">
</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>Tom</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Dec 17, 2015, at 2:39 PM, <<a moz-do-not-send="true" href="mailto:benoit.zuber@ana.unibe.ch" class=""></a><a class="moz-txt-link-abbreviated" href="mailto:benoit.zuber@ana.unibe.ch">benoit.zuber@ana.unibe.ch</a>>
<<a moz-do-not-send="true" href="mailto:benoit.zuber@ana.unibe.ch" class="">benoit.zuber@ana.unibe.ch</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode:
space; -webkit-line-break: after-white-space; font-size:
14px; font-family: Calibri, sans-serif;" class="">
<div class="">
<div class="">Hi Tom</div>
<div class=""><br class="">
</div>
<div class="">Thanks for your thorough reply. I am not
in front of the workstation but I’ll use the gpu
activity monitor to guess if chimera is using the
three GPUs. I have some doubts that the OS / openGL
driver is distributing rendering work on each gpu in
our configuration. I think that for the SLI to work
you need to wire both cards together, which is not
the case in our workstation. But anyway, I’ll check
activity.</div>
<div class=""><br class="">
</div>
<div class="">The rendering case I was referring to
consists of solid rendering. I never try to render
the full dataset at full resolution as we are
talking about datasets of around 50 Gb. I use a very
high step to select a sub region, then I gradually
decrease step size and fine tune the contrast
transfer function. When I am happy I render at step
2 or step 1, depending on the sub region size.
Clearly there is time needed to read data from disk;
when reading is done the program still computes
stuff for a few seconds. Then I can rotate, zoom, …,
the volume. If the sub region is not too large then
it is super smooth. But if I select a too large sub
volume, then I can rotate some degrees, then it
freezes for a while, then it is smooth again, etc.
So I guess it corresponds to your description of the
problem. </div>
<div class=""><br class="">
</div>
<div class="">I took a look at the benchmark. That’s
very useful! Obviously I am not going to get better
performance with a quadro (at least one in the
benchmark: All quadro except K4000 get worse scores
for solid rendering than the GTX 780 Ti. K4000
actually got the same score as the 780 Ti. Only with
a GTX Titan would I get a nearly 2x improvement,
which makes sense since this card has 2x the memory
of the 780 Ti (6Gb vs 3Gb). The GTX Titan X is not
in the benchmark yet but it should be even better as
it has 12 Gb Ram. Possibly the quadro k6000, which
is not in the benchmark either, could have similar
or better performance as this one (it also has 12 Gb
RAM) but … it costs about 4 times more than a Titan
X. Sad that only the quadro work for stereo under
linux (but for that we bought a once state of the
art quadro on eBay for 150$ and installed it on
another PC :-)</div>
<div class=""><br class="">
</div>
<div class="">Cheers </div>
<div class="">Ben</div>
<div class="">
</div>
</div>
<div class=""><br class="">
</div>
<span id="OLK_SRC_BODY_SECTION" class="">
<div style="font-family: Calibri; font-size: 12pt;
text-align: left; border-width: 1pt medium medium;
border-style: solid none none; padding: 3pt 0in 0in;
border-top-color: rgb(181, 196, 223);" class="">
<span style="font-weight:bold" class="">De : </span>Tom
Goddard <<a moz-do-not-send="true" href="mailto:goddard@sonic.net" class="">goddard@sonic.net</a>><br class="">
<span style="font-weight:bold" class="">Date : </span>jeudi,
17 décembre 2015 19:22<br class="">
<span style="font-weight:bold" class="">À : </span>Benoit
Zuber <<a moz-do-not-send="true" href="mailto:benoit.zuber@ana.unibe.ch" class="">benoit.zuber@ana.unibe.ch</a>><br class="">
<span style="font-weight:bold" class="">Cc : </span>chimera
users list <<a moz-do-not-send="true" href="mailto:chimera-users@cgl.ucsf.edu" class="">chimera-users@cgl.ucsf.edu</a>><br class="">
<span style="font-weight:bold" class="">Objet : </span>Re:
[Chimera-users] multi GPU rendering<br class="">
</div>
<div class=""><br class="">
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0
5; MARGIN:0 0 0 5;" class="" type="cite">
<div class="">
<div style="word-wrap: break-word;
-webkit-nbsp-mode: space; -webkit-line-break:
after-white-space;" class="">
Hi Ben,
<div class=""><br class="">
</div>
<div class=""> I haven’t tried Chimera with 2
or more graphics cards working together. I
don’t know if it is likely to improve
rendering speed. I don’t think applications
need any special support to use multiple cards
— the system graphics driver distributes the
work across 2 or more cards. With Nvidia
cards this technology is called SLI and with
AMD cards it is called CrossFire. From what I
understand they do thing like have each of 2
cards render every other frame — since the
graphics are pipelined the current frame and
next frame could be rendered simultaneously
possibly doubling the speed. I have my doubts
that this will improve Chimera rendering speed
for large density maps. I think slow
rendering of maps in solid (grayscale) style
typically happens in Chimera when the map data
doesn’t fit on the graphics card, so every
frame it has to transfer all the data to the
card — the performance plummets once you get a
map that big. If the data does fit on the
graphics card, it usually renders at full
frame rate (60 frames per second), although
that isn’t true of all graphics cards. This
all relates to the speed rotating the model.
When you first load a big map it can take a
long time because disk drive speed is slow — a
solid state drive helps speed this up. If you
explain exactly the case where you see slow
rendering I can perhaps advise.</div>
<div class=""><br class="">
</div>
<div class=""> In general I think the Geforce
cards perform better (fewer bugs, faster
speed) than the Quadro cards, and would only
recommend Quadro if you use stereoscopic
display with shutter glasses. Here are
Chimera benchmarks for a range of graphics
cards:</div>
<div class=""><br class="">
</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span><a moz-do-not-send="true" href="http://plato.cgl.ucsf.edu/trac/chimera/wiki/benchmarks" class=""></a><a class="moz-txt-link-freetext" href="http://plato.cgl.ucsf.edu/trac/chimera/wiki/benchmarks">http://plato.cgl.ucsf.edu/trac/chimera/wiki/benchmarks</a></div>
<div class=""><br class="">
</div>
<div class=""> Tom</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Dec 17, 2015, at 5:28 AM,
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word;
-webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;
font-size: 14px; font-family: Calibri,
sans-serif;" class="">
<div class="">Hi</div>
<div class=""><br class="">
</div>
<div class="">Can chimera make use of
more than one GPU for 3D rendering?
We have 3 Geforce GTX 780 Ti (mainly
used for number crunching) on a
particular workstation and would be
interested to make full use of them
for rendering tomograms or serial
block face imaging data.</div>
<div class=""><br class="">
</div>
<div class="">Another question: has
anyone done benchmark comparison for
3D rendering in Chimera between a
top of the line Geforce GTX card and
a top Quadro card? Is there a strong
improvement with the Quadro, which
might justify the price gap? </div>
<div class=""><br class="">
</div>
<div class="">Thanks</div>
<div class="">Ben</div>
<div class="">
<div id="" class="">
<div class="">
<div class="">-- </div>
<div class="">Prof. Benoît Zuber</div>
</div>
<div class="">Institute of Anatomy</div>
<div class="">University of Bern</div>
<div class="">Baltzerstrasse 2</div>
<div class="">Postfach 922</div>
<div class="">3000 Bern 9</div>
<div class="">Switzerland</div>
<div class=""><a moz-do-not-send="true" href="mailto:benoit.zuber@ana.unibe.ch" class=""></a><a class="moz-txt-link-abbreviated" href="mailto:benoit.zuber@ana.unibe.ch">benoit.zuber@ana.unibe.ch</a></div>
<div class="">+41 31 631 84 40</div>
<div class=""><a moz-do-not-send="true" href="http://www.ana.unibe.ch/research/experimental_morphology/index_eng.html" class=""></a><a class="moz-txt-link-freetext" href="http://www.ana.unibe.ch/research/experimental_morphology/index_eng.html">http://www.ana.unibe.ch/research/experimental_morphology/index_eng.html</a></div>
</div>
</div>
</div>
_______________________________________________<br class="">
Chimera-users mailing list<br class="">
<a moz-do-not-send="true" href="mailto:Chimera-users@cgl.ucsf.edu" class="">Chimera-users@cgl.ucsf.edu</a><br class="">
<a moz-do-not-send="true" href="http://plato.cgl.ucsf.edu/mailman/listinfo/chimera-users" class="">http://plato.cgl.ucsf.edu/mailman/listinfo/chimera-users</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</span>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<br class="">
</div>
</div></blockquote></div><br class=""></div></body></html>