Uploaded byHBaseCon
HBaseCon 2015: Blackbird Collections - In-situ Stream Processing in HBase
The document discusses the implementation of Blackbird collections for in-situ stream processing within HBase at Rocketfuel Inc., focusing on the challenges faced such as memstore bloat and concurrency bugs. It outlines a logical model for collections that allows for the addition of elements and applies various functions during reads and normalization processes to maintain data integrity. The document also presents examples of different collection types, including segment sets and transient counters, along with their operational methodologies and performance optimizations.
More Related Content
Similar to HBaseCon 2015: Blackbird Collections - In-situ Stream Processing in HBase
HBaseCon 2015: Blackbird Collections - In-situ Stream Processing in HBase
- 1. Blackbird Collections In-situ StreamProcessing in HBase Ishan Chhabra, Nitin Aggarwal, Deepankar Reddy Rocketfuel Inc.
- 2. Quick Recap: Blackbird @Rocketfuel Inc.
- 3. Quick Recap: Blackbird @Rocketfuel Inc.
- 4. Collections: At the centerof all non-trivial applications
- 5. Usual first attempt: ReadModify Write Read existing 100 elementsEvent Write 101 elements
- 6. Problem: Not sympathetic to HBaseInternals Memstore bloat Constant flushing Compactions
- 7. Problem: Asymptotic network usage DC 1DC 2 Write 1: 101 elements Write 2: 48 elements
- 8. Problem: Concurrency Bug Readexisting 100 elements Event 1 Write 101 elements 100 elements Read existing 100 elements Write 101 elements Event 2 101 elements
- 9. Append Only Collections: Besympathetic to HBase Internals Trivial example: Lists But what about Sets, Maps, Counters, etc. and domain specific collections?
- 10.
- 11. Working example: SegmentSet Keep thelatest entry only for a segment Atmost 1000 of the most recently updated segments
- 12. Blackbird Collections: Logical Model Collection ofentries Can only add elements to it Apply series of functions during read to enforce properties
- 13. Blackbird Collections: Logical Model For everycollection: Define the structure Define the series of functions f1, f2, f3… to apply during read
- 14. So how would SegmentSetwork? Initial State in DB
- 15. So how would SegmentSetwork? Adding a new element in DB
- 16. So how would SegmentSetwork? Enforce properties during read at client
- 17.
- 18. Combined Column: 100entries views:4587 1 entry views:2398 Logical Collection 2 entries views:6798 1 entry views:2983 Step 1: Write appends to separate columns
- 19. Step 2: Applythe functions during reads Combined Column: 100 entries 1 entry 2 entries 1 entry 104 entries 92 entries f1, f2, …
- 20. But what aboutall the garbage that is building up?
- 21. Step 3: Normalization Combined Column:100 entries 1 entry 2 entries 1 entry 92 entries
- 22. Step 3: Normalization 2 kindsof runs: nightly and weekly Nightly run only looks at subset of data (data changed that day) Weekly run looks at all the data
- 23. Step 3: Normalization Heavily optimized:< 1h for nightly run and 2- 3h for weekly run (~50TB of data) Made fast by MR over snapshots and bulkloads No impact on live read performance
- 24. Blackbird Collections: Updated LogicalModel Collection of entries Can only add elements to it Apply series of functions during reads Apply series of functions during daily normalizationApply series of functions during weekly normalization
- 25.
- 26. Another Example: Transient Counters Beable to increment/decrement counts Remove entries if timestamp + time to live < current time Keep the latest 1000 entries only
- 27.
- 28. Another Example: Transient Counters aggregate()expire()During read: aggregate() expire() limit_to_1000() Daily Normalization: aggregate() expire() limit_to_1000() Weekly Normalization:
- 29.
- 30. Thank you! Questions? Reach usat: ishan@rocketfuel.com naggarwal@rocketfuel.com dreddy@rocketfuel.com
