[OpenWalnut-Dev] Why OW is slow as hell loading datasets and uses almost 2gig of ram on a 600mb dataset

[Sebastian Eichelbaum]

We knew the problem. There are lot of unneeded copy operations involved in the whole data handling stuff. Me and Andre are working on a completely new data handling framework. So a re-factoring of the old code quite unlikely.

Thank you your effort. There are several other points where a refactoring might be a good idea.

Bye and a nice weekend too.

On Fri, 08 Oct 2010, Robin Ledig wrote:

>  Hi everyone,
> 
> after searching for the reason that OW crashed for me while loading
> a huge dataset with an exception of bad allocation i took the
> liberty to look into that.
> The result is that the loader is extremly inefficent in terms of
> memory and speed.
> Here is why.
> First lets have a look at
> 
> boost::shared_ptr< WDataSet > WReaderNIfTI::load()
> in src\dataHandler\io\WReaderNIfTI.cpp
> 
> here we got a temporary pointer
> nifti_image* filedata = nifti_image_read( m_fname.c_str(), 1 );
> holding the loaded data of the nifti file, so in my case 600mb
> 
> what happens next?
> The data is rearenged to fit the needs of a WValueSet with a call to
> WReaderNIfTI::copyArray
> Which is not that bad so far.
> 
> So lets have a closer look at  copyArray
> 
> here a vector with almost the same size as the nifti data is
> allocated, so again 600mb of data (1,2gb right here)
> std::vector< T > data( countVoxels * vDim );
> 
> now the data gets rearenged
> 
> here comes a little thing i don't like:
> return data;
> 
> Why so?
> Lets have a look at the call of copyArray:
> std::vector< T > data = copyArray( reinterpret_cast< T* >(
> filedata->data ), countVoxels, vDim );
> 
> this means that the copyArrays return value's copy constructur is
> called, which in the case of std::vector means a complete copy is
> constructed.
> So in my case i at this point i need temporarly 1,8gb ram with 3x
> 600mb of conncted memory.
> After that copyConstructor call i get 600mb free again. Yeah back
> 1,2gb used memory.
> Whats next.
> Oh and again
> 
> newValueSet = boost::shared_ptr< WValueSetBase >( new WValueSet< T
> >( order, vDim, data, W_DT_SIGNED_INT ) );
> a constructor call with the argument of a std::vector so again a
> copy call of my 600mb.
> 
> So the overall data transfer for creating the new valueset is as follows
> - reading data from disc (600mb) which is necessary
> - rearange data to fit to WValueSet needs (+600mb) also seams reasonable
> - copy data from copyArray to "data" (+600mb) not necessary
> - copy data from "data" to newValueSet (+600mb) not necessary
> 
> so overall we have a transferVolume of 2,4gb of data for loading a
> 600mb dataset with a temporary memory peak of 1,8gb.
> 
> You might consider this for a next meeting to Refactor some basic
> code structures.
> 
> Cheers and a nice weekend to all of you.
> Robin
> _______________________________________________
> OpenWalnut-Dev mailing list
> OpenWalnut-Dev at lists.informatik.uni-leipzig.de
> http://lists.informatik.uni-leipzig.de/mailman/listinfo/openwalnut-dev
> 

-- 
Dipl.-Inf. Sebastian Eichelbaum
Universität Leipzig
Institut für Informatik
Abteilung Bild- und Signalverarbeitung
PF 100920
D-04009 Leipzig