Post Reply 
 
Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
design a DSPSs for Data Thinker
01-27-2015, 11:53 PM
Post: #1
design a DSPSs for Data Thinker
Recently, I am studying MillWheel, and I have an idea to build a DSPSs(Distributed Stream Processing Systems) for Data Thinker. Honestly, I have not fully ability to make it solely,so I wish anyone who can give me a five.

Before start to design a such system, It is necessary to know what I need to care about. I list something what I consider at below:

1.recover mechanism:once a node failed, the backup should take over that node to process continually.
2.persistent state: store metadata for recover or anything else.
3.handle out-of-order data: make sure data are consistency.
4.fault tolerance mechanism: for delivery or process when a node failed.

If I miss something, please point it out. Thanks.
Quote this message in a reply
01-31-2015, 03:07 PM
Post: #2
RE: design a DSPSs for Data Thinker
About recover mechanism:
I found high-Availability algorithms in Hwang†:High-Availability Algorithms for Distributed Stream Processing
In the paper, it describe three approach to recover, they offer different tradeoffs between runtime overhead and recovery performance.
Quote this message in a reply
02-10-2015, 02:04 PM
Post: #3
RE: design a DSPSs for Data Thinker
The SPE is base of DSPSs, so I try to design a SPE first. refer to http://tab.d-thinker.org/showthread.php?tid=3125
Quote this message in a reply
Post Reply 


Forum Jump: