[OpenWalnut-Dev] Why OW is slow as hell loading datasets and uses almost 2gig of ram on a 600mb dataset

[Robin Ledig]

  Hi everyone,

after searching for the reason that OW crashed for me while loading a 
huge dataset with an exception of bad allocation i took the liberty to 
look into that.
The result is that the loader is extremly inefficent in terms of memory 
and speed.
Here is why.
First lets have a look at

boost::shared_ptr< WDataSet > WReaderNIfTI::load()
in src\dataHandler\io\WReaderNIfTI.cpp

here we got a temporary pointer
nifti_image* filedata = nifti_image_read( m_fname.c_str(), 1 );
holding the loaded data of the nifti file, so in my case 600mb

what happens next?
The data is rearenged to fit the needs of a WValueSet with a call to 
WReaderNIfTI::copyArray
Which is not that bad so far.

So lets have a closer look at  copyArray

here a vector with almost the same size as the nifti data is allocated, 
so again 600mb of data (1,2gb right here)
std::vector< T > data( countVoxels * vDim );

now the data gets rearenged

here comes a little thing i don't like:
return data;

Why so?
Lets have a look at the call of copyArray:
std::vector< T > data = copyArray( reinterpret_cast< T* >( 
filedata->data ), countVoxels, vDim );

this means that the copyArrays return value's copy constructur is 
called, which in the case of std::vector means a complete copy is 
constructed.
So in my case i at this point i need temporarly 1,8gb ram with 3x 600mb 
of conncted memory.
After that copyConstructor call i get 600mb free again. Yeah back 1,2gb 
used memory.
Whats next.
Oh and again

newValueSet = boost::shared_ptr< WValueSetBase >( new WValueSet< T >( 
order, vDim, data, W_DT_SIGNED_INT ) );
a constructor call with the argument of a std::vector so again a copy 
call of my 600mb.

So the overall data transfer for creating the new valueset is as follows
- reading data from disc (600mb) which is necessary
- rearange data to fit to WValueSet needs (+600mb) also seams reasonable
- copy data from copyArray to "data" (+600mb) not necessary
- copy data from "data" to newValueSet (+600mb) not necessary

so overall we have a transferVolume of 2,4gb of data for loading a 600mb 
dataset with a temporary memory peak of 1,8gb.

You might consider this for a next meeting to Refactor some basic code 
structures.

Cheers and a nice weekend to all of you.
Robin