Downloaded 60 times
![$0.56
$2.42
1.25
$2.11
$1.26
$2.78
$0.756
$0.809
$2.42
1.25
$2.11
$1.26
$2.78
$1.256
$1.809
$2.42
1.25
$2.11
$1.26
Site/PageGeo/WeatherTime of DayBrand AffinityUser
[ + ][ + ]](https://image.slidesharecdn.com/casestudies-session2-140616154221-phpapp02/85/Case-studies-session-2-3-320.jpg)
![» In absence of proxy: ‘The client is part of the cluster’ [1]
» Client must report availability error to calling application thread in short time span
» Follow circuit breaker pattern for read calls (Anecdote)
» ‘pseudo’ puts (local file) for write calls
[1] Blog post from Lars Hofhansl http://hadoop-hbase.blogspot.com/2012/09/hbase-client-timeouts.html](https://image.slidesharecdn.com/casestudies-session2-140616154221-phpapp02/85/Case-studies-session-2-40-320.jpg)
Uploaded byHBaseCon
Case studies session 2
Blackbird is Rocket Fuel's real-time bidding platform that processes billions of rows and bid requests in milliseconds. It uses HBase for its data store and has implemented various optimizations to achieve high performance, such as using protocol buffers, caching, append-only writes, and off-peak compactions. It also focuses on reliability through techniques like aggressive monitoring, dynamic blacklisting, and handling client failures without relying on a proxy server.
More Related Content
Similar to Case studies session 2
Case studies session 2
- 1. Blackbird Billions of rows,couple of milliseconds away Ishan Chhabra Shrijeet Paliwal Abhijit Pol
- 3.
- 5. User Segments 3. Bid Request 5.Rocket Fuel Winning Ad 2. Ad Request 6. Ad Served 1. Page Request 4. Bid & Ad Browser User Engagements Publishers Data Partners Exchange Partners Optimize Simple View of Rocket Fuel Platform Real-time Bidder User Engagement User Data Store Model Scoring
- 6. So what isBlackbird?
- 7. 5 B 6 B 45B Facebook likes Searches on Google Bid Requests Considered by Rocketfuel Requests per day
- 8. 400 100 20 2 Blink of aneye SF to Tokyo network round trip One beat of a hummindbird's wing Look up in Blackbird Time (ms)
- 9.
- 10. HBase, we havea problem..
- 11.
- 12.
- 13.
- 14. » Data losson concurrent modification » Read per write » High amount of data per write » O(n)
- 15. » Significantly reducedflushes, compaction, network usage, GC. » O(1)
- 16. Combined Column: 100entries c1:combined 1 entry c1:rand1 2 entries c1:rand2 1 entry c1:rand3 Logical Collection Append Only, the HBase view
- 17. Optimizing reads usingnormalization Combined Column: 100 entries c1:combined 1 entry c1:rand1 2 entries c1:rand2 1 entry c1:rand3 Combined Column: 103 entries c1:combined
- 18.
- 19.
- 20.
- 21. Not so easy! HBaseis designed for high throughput writes
- 22. Key Ideas Read as littleas possible Stay stable, uniform, data local Don’t go to disk Even if you have to go to disk, make it fast
- 23.
- 24. Stay stable, uniform,data local at all times Good quality hardware Properly designed row keys Off peak daily major compaction
- 25. Give me allyour Cache! 128 Gb machines with 50% block Cache High Cache hit ratio (90% +) by effective utilization
- 27. It’s time todisk(o) 15K SAS drives Local & Short circuit reads (20-30% improvement)
- 28. High throughput writesare supported too!
- 29. Small Writes • AppendOnly • Protobufs Large Memstores • 4 Gb • Avoids flushes, memory churn, compaction • Maintains read performance by avoiding multiple seeks Tuned Compaction • Avoid Minor compactions • Off Peak Major compaction
- 30.
- 31. Organize the chaosor pay the cost..
- 32. » Blind writescan grow rows & table too big » Newbie clients 'guess' a lot » Simple queries such as row count can be hard on the fly Be aware…
- 33. Web app BidServing Ad servingData augmentation Batch data pipelines Ops Housekeeping Real time data pipelines Multitenant Blackbird
- 34. Multi tenancy makesit hard to find the defaulter Use ACLs & client side metrics in all access paths
- 35. Draft guidelines fornew clients, help them estimate the growth Keep track of growth, row count, row size, column size etc.
- 36.
- 37. It’s a delicateequilibrium that is hard to maintain Shield it with aggressive alerting, dashboards & canary monitoring
- 38. 1st region serverdies after several hours of clogged RPC queue Bad region moves to another region server & soon kills it too! 2jmj7l5rSw0yVb_vlWAYkK_Ybwk stgLVlK_SsLMn4HoG82ymp-QlRtA Clients can go rouge, it can get as bad as a DoS attack Protection via dynamic blacklists & size limit filters
- 39.
- 40. » In absenceof proxy: ‘The client is part of the cluster’ [1] » Client must report availability error to calling application thread in short time span » Follow circuit breaker pattern for read calls (Anecdote) » ‘pseudo’ puts (local file) for write calls [1] Blog post from Lars Hofhansl http://hadoop-hbase.blogspot.com/2012/09/hbase-client-timeouts.html
- 41.
- 42. Obligatory “we arehiring” slide! http://rocketfuel.com/careers ishan@rocketfuel.com shrijeet@rocketfuel.com apol@rocketfuel.com
