[OpenWalnut-Dev] Buffered WModuleInputConnector

[Christof Pieloth]

Hey Mario,

"so the problem is that one module is streaming data maybe faster than the client can process the data but you must process all data by the client module, e.g., to store the data in a file?"

Yes.

"What are the "data conflicts" you mention? If they are skipped data (let's call them frames), then this is intended and I would not call it a conflict. If it leads to a crash or invalid data, then it indeed is a problem and we should rethink the synchronization."

There are no conflicts like segmentation fault or similar. It is just on 
the logical level of the application, so frames should not be skipped 
or/and if frames are skipped it can be noticed.

Regards,
Christof

Am 30.05.2012 11:51, schrieb Mario Hlawitschka:
> Hi Christof,
>
> so the problem is that one module is streaming data maybe faster than the client can process the data but you must process all data by the client module, e.g., to store the data in a file?
> Then it sounds reasonable to me.
>
> What are the "data conflicts" you mention? If they are skipped data (let's call them frames), then this is intended and I would not call it a conflict. If it leads to a crash or invalid data, then it indeed is a problem and we should rethink the synchronization.
>
> 	Mario
>
>
> On May 30, 2012, at 10:06 AM, Christof Pieloth wrote:
>
>> Hi,
>>
>> Alex is right. We are working on  real-time analysis of EEG and MEG signals. In a use case the first module is streaming some data (a connected EEG device or a loaded FIFF file (simulation)) and connected modules are processing the incoming data. So, in your module graph you can build a sequential or parallel pipeline of modules on top of the streaming module. Each module implements some algorithms like preprocessing (FIR filter) or trigger detection.
>> Because the computing time depends on the algorithm and data, a module requires a buffer for incomming data to prevent some data conflicts.
>>
>> Let's have a chat in IRC.
>>
>> Bye,
>> Christof
>>
>>
>> Am 29.05.2012 19:03, schrieb Alexander Wiebel:
>>> Hi Mario,
>>> ich vermute es geht ums Echtzeit-EEG, dass ich 2010 mit Mirco Fuchs als
>>> DA ausgeschrieben hatte (siehe Anhang).
>>>
>>> Gruß Alex
>>>
>>> On 05/29/2012 06:42 PM, Mario Hlawitschka wrote:
>>>> Hi Christof,
>>>>
>>>> thanks for contributing to OW.
>>>>
>>>> Not knowing what you are working on, could you explain in a bit more detail what the queue/ring buffer etc. should do in the context of input and output connectors? Do you have an example case where you are using it to demonstrate its purpose? Maybe one, that could serve as part of the OW  user documentation as well.
>>>>
>>>>
>>>> Looking at the diff, there are some minor language issues in the documentation, e.g.:
>>>> This class checks whether the input connector is an instance of WModuleInputDataCollection, if is it true. =>   More or less incomplete sentence?
>>>> We can discuss those off-list.
>>>>
>>>> Is there a reason why thread-safety is only guaranteed for one producer and once consumer in the ring buffer implementation?
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> 	Mario
>>>>
>>>>
>>>> On May 29, 2012, at 5:50 PM, Christof Pieloth wrote:
>>>>
>>>>> Hello OW developers,
>>>>>
>>>>> we implement an interface and class to realize a buffered or collection-like WModuleInputConnector. This should be useful for data processing through a pipeline of modules.
>>>>> Each module can use their own implementation of "WModuleInputDataCollection" like a queue, ring buffer or more. The source module must use "WModuleOutputDataCollectionable" to fill the connected WModuleInputDataCollections, because WModuleInputData only fetches the data from WModuleOutputData.
>>>>> It would be great if the diff, or an improved diff, could be included in default.
>>>>>
>>>>> Bye,
>>>>> Christof Pieloth
>>>>>
>>>>> -- 
>>>>> HTWK Leipzig, University of Applied Sciences
>>>>> Faculty of Electrical Engineering and Information Technology
>>>>> Laboratory for Biosignal Processing (LaBP)
>>>>>
>>>>> fon:  +49 (0) 341 3076 3133
>>>>> fax:  +49 (0) 341 3076 853133
>>>>> mail: pieloth 'at' labp.htwk-leipzig.de
>>>>>
>>>>>
>>>>> <InputOutput.diff>_______________________________________________
>>>>> OpenWalnut-Dev mailing list
>>>>> OpenWalnut-Dev at lists.informatik.uni-leipzig.de
>>>>> http://lists.informatik.uni-leipzig.de/mailman/listinfo/openwalnut-dev
>>>>>
>>>>> Archive: http://lists.informatik.uni-leipzig.de/pipermail/openwalnut-dev/
>>>> _______________________________________________
>>>> OpenWalnut-Dev mailing list
>>>> OpenWalnut-Dev at lists.informatik.uni-leipzig.de
>>>> http://lists.informatik.uni-leipzig.de/mailman/listinfo/openwalnut-dev
>>>>
>>>> Archive: http://lists.informatik.uni-leipzig.de/pipermail/openwalnut-dev/
>> -- 
>> HTWK Leipzig, University of Applied Sciences
>> Faculty of Electrical Engineering and Information Technology
>> Laboratory for Biosignal Processing (LaBP)
>>
>> fon:  +49 (0) 341 3076 3133
>> fax:  +49 (0) 341 3076 853133
>> mail: pieloth 'at' labp.htwk-leipzig.de
>>
>>
>> _______________________________________________
>> OpenWalnut-Dev mailing list
>> OpenWalnut-Dev at lists.informatik.uni-leipzig.de
>> http://lists.informatik.uni-leipzig.de/mailman/listinfo/openwalnut-dev
>>
>> Archive: http://lists.informatik.uni-leipzig.de/pipermail/openwalnut-dev/
>>
> _______________________________________________
> OpenWalnut-Dev mailing list
> OpenWalnut-Dev at lists.informatik.uni-leipzig.de
> http://lists.informatik.uni-leipzig.de/mailman/listinfo/openwalnut-dev
>
> Archive: http://lists.informatik.uni-leipzig.de/pipermail/openwalnut-dev/

-- 
HTWK Leipzig, University of Applied Sciences
Faculty of Electrical Engineering and Information Technology
Laboratory for Biosignal Processing (LaBP)

fon:  +49 (0) 341 3076 3133
fax:  +49 (0) 341 3076 853133
mail: pieloth 'at' labp.htwk-leipzig.de