[Chimera-users] binning large volumes

[Tom Goddard]

Hi Lori,

  If you had enough memory the resample operation would be pretty fast 
(I'd guess a few tens of seconds).  But the trouble is that Chimera is 
doing this operation in a very memory inefficient way.  What it does is 
determine the x,y,z coordinates of all the grid points and then 
interpolates the volume at those points.  The trouble is that those 
coordinates are 3 floats taking 12 bytes.  Also it makes a floating 
point copy of the interpolated values for another 4 bytes per grid 
point. If your 2 Gbyte map is single byte values then these extra arrays 
take 16 times more memory, 32 Gbytes.  If your maps are 4 byte floats 
you still use 4 times the memory that the map does (8 Gbytes).   If this 
stuff doesn't fit in memory it starts getting paged to disk and it may 
be 100 times slower.

  The only reasonable answer here is that Chimera should not allocate 
memory for all the xyz grid points.  It's doing that because the code is 
in Python/numpy instead of C++.  Other code in Chimera instead works one 
data plane at a time.  This reduces the temporary memory use for those 
xyz grid points and if the whole calculation can be done in memory it is 
much much faster.  I just made that change for resampling and it will be 
in tomorrow's daily build.  It won't be in the upcoming Chimera 1.4 
release since are only making bug fixes to that version.  The memory use 
is still quite high.  It makes an extra temporary copy of the 
interpolated map with floating point values (4 bytes per grid point).  
That is unnecessary.  But it's a bit more trouble to optimize that away 
so I'm not going to do it now.  But I'll add it to the Chimera request 
list.  With the change I made it should use 4 bytes per grid point 
temporary memory instead of 16 bytes and if your 64-bit Linux system has 
enough physical memory (8 or 16 Gbytes) the resample will run much faster.

    Tom


-------- Original Message --------
Subject: [Chimera-users] binning large volumes
From: Lori Nyland
To: chimera-users
Date: 10/28/09 1:07 PM
> Hello,
>
> I have two volumes with the same dimensions and each are nearly 2GB. I 
> can use the vop resample command in a 64 bit Chimera for Linux and 
> reassign one volumes coordinates relative to the other. However, it is 
> slow to execute, but it does work.
> In order to speed things up is there a way to bin the volumes (to use 
> less space), place it through the vop resample command, and then 
> un-bin the volume without losing information?
>
> Thanks!
> Lori