[OpenWalnut-Dev] Buffered WModuleInputConnector

Tue Jun 5 14:51:01 CEST 2012

Hi Christof

Thank you once again for your contribution. Unfortunately, I will not have the time to check your patch today (hopefully at the end of the week). 

I will contact you again once I had a look at your code.
Thank you an best wishes
Sebastian

On Wed, 30 May 2012, Christof Pieloth wrote:

> We are using OW on both ways. We are doing data processing (main
> part) and visualize the processed data. Between the modules we are
> transfering data packets (no frames/images) and each module may, but
> not need, visualize these data. And you're right, on the
> visualization part we do not have to take every packet, but we need
> to for the underlying processing part.
> 
> I don't know each module of OW, but  I thought it could be useful
> for modules which are doing similar things. For example, some
> clustering algorithms in which the processing time depends not only
> on the input parameters, but also on the input data, even withsame
> data size. In such a caseall images could be processed.
> 
> See you
> Christof
> 
> 
> Am 30.05.2012 15:58, schrieb Mario Hlawitschka:
> >Hi Christoph,
> >
> >I agree with the second bullet point: We should be able to find out whether frames are skipped and whether it is allowed to skip frames. In the sense of a visualization toolkit, there is no need to progress every single frame. In a data processing framework, this is required.
> >
> >I myself am streaming live data into the application and don't care about skipped frames as I want to see a live image of what is going on. But a flag notifying about skipped frames and some statistics would really be nice.
> >
> >Something to discuss at the next OW BBQ,
> >
> >         Mario
> >
> >
> >On May 30, 2012, at 2:05 PM, Christof Pieloth wrote:
> >
> >>Hey Mario,
> >>
> >>"so the problem is that one module is streaming data maybe faster than the client can process the data but you must process all data by the client module, e.g., to store the data in a file?"
> >>
> >>Yes.
> >>
> >>"What are the "data conflicts" you mention? If they are skipped data (let's call them frames), then this is intended and I would not call it a conflict. If it leads to a crash or invalid data, then it indeed is a problem and we should rethink the synchronization."
> >>
> >>There are no conflicts like segmentation fault or similar. It is just on the logical level of the application, so frames should not be skipped or/and if frames are skipped it can be noticed.
> >>
> >>Regards,
> >>Christof
> >>
> >>Am 30.05.2012 11:51, schrieb Mario Hlawitschka:
> >>>Hi Christof,
> >>>
> >>>so the problem is that one module is streaming data maybe faster than the client can process the data but you must process all data by the client module, e.g., to store the data in a file?
> >>>Then it sounds reasonable to me.
> >>>
> >>>What are the "data conflicts" you mention? If they are skipped data (let's call them frames), then this is intended and I would not call it a conflict. If it leads to a crash or invalid data, then it indeed is a problem and we should rethink the synchronization.
> >>>
> >>>      Mario
> >>>
> >>>
> >>>On May 30, 2012, at 10:06 AM, Christof Pieloth wrote:
> >>>
> >>>>Hi,
> >>>>
> >>>>Alex is right. We are working on  real-time analysis of EEG and MEG signals. In a use case the first module is streaming some data (a connected EEG device or a loaded FIFF file (simulation)) and connected modules are processing the incoming data. So, in your module graph you can build a sequential or parallel pipeline of modules on top of the streaming module. Each module implements some algorithms like preprocessing (FIR filter) or trigger detection.
> >>>>Because the computing time depends on the algorithm and data, a module requires a buffer for incomming data to prevent some data conflicts.
> >>>>
> >>>>Let's have a chat in IRC.
> >>>>
> >>>>Bye,
> >>>>Christof
> >>>>
> >>>>
> >>>>Am 29.05.2012 19:03, schrieb Alexander Wiebel:
> >>>>>Hi Mario,
> >>>>>ich vermute es geht ums Echtzeit-EEG, dass ich 2010 mit Mirco Fuchs als
> >>>>>DA ausgeschrieben hatte (siehe Anhang).
> >>>>>
> >>>>>Gruß Alex
> >>>>>
> >>>>>On 05/29/2012 06:42 PM, Mario Hlawitschka wrote:
> >>>>>>Hi Christof,
> >>>>>>
> >>>>>>thanks for contributing to OW.
> >>>>>>
> >>>>>>Not knowing what you are working on, could you explain in a bit more detail what the queue/ring buffer etc. should do in the context of input and output connectors? Do you have an example case where you are using it to demonstrate its purpose? Maybe one, that could serve as part of the OW  user documentation as well.
> >>>>>>
> >>>>>>
> >>>>>>Looking at the diff, there are some minor language issues in the documentation, e.g.:
> >>>>>>This class checks whether the input connector is an instance of WModuleInputDataCollection, if is it true. =>    More or less incomplete sentence?
> >>>>>>We can discuss those off-list.
> >>>>>>
> >>>>>>Is there a reason why thread-safety is only guaranteed for one producer and once consumer in the ring buffer implementation?
> >>>>>>
> >>>>>>
> >>>>>>Thanks,
> >>>>>>
> >>>>>>   Mario
> >>>>>>
> >>>>>>
> >>>>>>On May 29, 2012, at 5:50 PM, Christof Pieloth wrote:
> >>>>>>
> >>>>>>>Hello OW developers,
> >>>>>>>
> >>>>>>>we implement an interface and class to realize a buffered or collection-like WModuleInputConnector. This should be useful for data processing through a pipeline of modules.
> >>>>>>>Each module can use their own implementation of "WModuleInputDataCollection" like a queue, ring buffer or more. The source module must use "WModuleOutputDataCollectionable" to fill the connected WModuleInputDataCollections, because WModuleInputData only fetches the data from WModuleOutputData.
> >>>>>>>It would be great if the diff, or an improved diff, could be included in default.
> >>>>>>>
> >>>>>>>Bye,
> >>>>>>>Christof Pieloth
> >>>>>>>
> >>>>>>>--
> >>>>>>>HTWK Leipzig, University of Applied Sciences
> >>>>>>>Faculty of Electrical Engineering and Information Technology
> >>>>>>>Laboratory for Biosignal Processing (LaBP)
> >>>>>>>
> >>>>>>>fon:  +49 (0) 341 3076 3133
> >>>>>>>fax:  +49 (0) 341 3076 853133
> >>>>>>>mail: pieloth 'at' labp.htwk-leipzig.de
> >>>>>>>
> >>>>>>>
> >>>>>>><InputOutput.diff>_______________________________________________
> >>>>>>>OpenWalnut-Dev mailing list
> >>>>>>>OpenWalnut-Dev at lists.informatik.uni-leipzig.de
> >>>>>>>http://lists.informatik.uni-leipzig.de/mailman/listinfo/openwalnut-dev
> >>>>>>>
> >>>>>>>Archive: http://lists.informatik.uni-leipzig.de/pipermail/openwalnut-dev/
> >>>>>>_______________________________________________
> >>>>>>OpenWalnut-Dev mailing list
> >>>>>>OpenWalnut-Dev at lists.informatik.uni-leipzig.de
> >>>>>>http://lists.informatik.uni-leipzig.de/mailman/listinfo/openwalnut-dev
> >>>>>>
> >>>>>>Archive: http://lists.informatik.uni-leipzig.de/pipermail/openwalnut-dev/
> >>>>--
> >>>>HTWK Leipzig, University of Applied Sciences
> >>>>Faculty of Electrical Engineering and Information Technology
> >>>>Laboratory for Biosignal Processing (LaBP)
> >>>>
> >>>>fon:  +49 (0) 341 3076 3133
> >>>>fax:  +49 (0) 341 3076 853133
> >>>>mail: pieloth 'at' labp.htwk-leipzig.de
> >>>>
> >>>>
> >>>>_______________________________________________
> >>>>OpenWalnut-Dev mailing list
> >>>>OpenWalnut-Dev at lists.informatik.uni-leipzig.de
> >>>>http://lists.informatik.uni-leipzig.de/mailman/listinfo/openwalnut-dev
> >>>>
> >>>>Archive: http://lists.informatik.uni-leipzig.de/pipermail/openwalnut-dev/
> >>>>
> >>>_______________________________________________
> >>>OpenWalnut-Dev mailing list
> >>>OpenWalnut-Dev at lists.informatik.uni-leipzig.de
> >>>http://lists.informatik.uni-leipzig.de/mailman/listinfo/openwalnut-dev
> >>>
> >>>Archive: http://lists.informatik.uni-leipzig.de/pipermail/openwalnut-dev/
> >>--
> >>HTWK Leipzig, University of Applied Sciences
> >>Faculty of Electrical Engineering and Information Technology
> >>Laboratory for Biosignal Processing (LaBP)
> >>
> >>fon:  +49 (0) 341 3076 3133
> >>fax:  +49 (0) 341 3076 853133
> >>mail: pieloth 'at' labp.htwk-leipzig.de
> >>
> >>
> >>_______________________________________________
> >>OpenWalnut-Dev mailing list
> >>OpenWalnut-Dev at lists.informatik.uni-leipzig.de
> >>http://lists.informatik.uni-leipzig.de/mailman/listinfo/openwalnut-dev
> >>
> >>Archive: http://lists.informatik.uni-leipzig.de/pipermail/openwalnut-dev/
> >>
> >_______________________________________________
> >OpenWalnut-Dev mailing list
> >OpenWalnut-Dev at lists.informatik.uni-leipzig.de
> >http://lists.informatik.uni-leipzig.de/mailman/listinfo/openwalnut-dev
> >
> >Archive: http://lists.informatik.uni-leipzig.de/pipermail/openwalnut-dev/
> 
> -- 
> HTWK Leipzig, University of Applied Sciences
> Faculty of Electrical Engineering and Information Technology
> Laboratory for Biosignal Processing (LaBP)
> 
> fon:  +49 (0) 341 3076 3133
> fax:  +49 (0) 341 3076 853133
> mail: pieloth 'at' labp.htwk-leipzig.de
> 
> 
> _______________________________________________
> OpenWalnut-Dev mailing list
> OpenWalnut-Dev at lists.informatik.uni-leipzig.de
> http://lists.informatik.uni-leipzig.de/mailman/listinfo/openwalnut-dev
> 
> Archive: http://lists.informatik.uni-leipzig.de/pipermail/openwalnut-dev/
> 

-- 
Dipl.-Inf. Sebastian Eichelbaum
Universität Leipzig
Institut für Informatik
Abteilung Bild- und Signalverarbeitung
PF 100920
D-04009 Leipzig