Uploaded byHBaseCon
HBaseCon 2015 General Session: The Evolution of HBase @ Bloomberg
This document discusses Bloomberg's use of HBase for its "medium data" needs in serving financial customers. It outlines challenges including speed, availability, and supporting many users. Bloomberg has invested in HBase to address these, seeing it as applicable to its "medium data" volumes. The document calls for continued work to further bolster HBase's reliability, take advantage of modern hardware, improve multi-tenancy, integrate Spark, and increase analytics efficiency. It expresses optimism for HBase's role in data platforms going forward.
More Related Content
Similar to HBaseCon 2015 General Session: The Evolution of HBase @ Bloomberg
HBaseCon 2015 General Session: The Evolution of HBase @ Bloomberg
- 1.
- 2. HBASEATBLOOMBERG// THE EVOLUTION OF BLOOMBERG DATASYSTEMSMEDIUM DATA NEEDS FOR THE FINANCIAL INDUSTRY MAY // 07 // 2015
- 3. HBASEATBLOOMBERG// BLOOMBERG 3 Leading Data andAnalytics provider to the financial industry
- 4. HBASEATBLOOMBERG// DATA IS OURBUSINESS 4
- 5. HBASEATBLOOMBERG// September 28: FullWorkshop at Bloomberg September 30: Showcase at Strata Hadoop Call for papers at: bloomberglabs.com/data-science DATA SCIENCE FOR SOCIAL GOOD: GOVERNMENT INNOVATION, PUBLIC HEALTH, ENVIRONMENT, EDUCATION
- 6. HBASEATBLOOMBERG// 6 • We havea “medium data” problem… • Speed and availability are paramount • Hundreds of thousands of users with expensive requests We’ve built many systems to address DATA MANAGEMENT TODAY
- 7. HBASEATBLOOMBERG// DATA MANAGEMENT CHALLENGES 7 •Single security analytics on Big Iron • Replication of Systems and Data • Complexity kills Top 500 Supercomputer list, 2013 >96% Linux. 100% of top 40.
- 8. HBASEATBLOOMBERG// DATA MANAGEMENT TOMORROW 8 •Simplicity and performance • Benefit from external developments • Retain our independence • Details matter
- 9. HBASEATBLOOMBERG// THE PREMISE 9 • Canapply big data techniques to our medium data problem, by addressing gaps in existing open systems • HBase is a good bet • Part of a broader whole • The Biggest community wins
- 10. HBASEATBLOOMBERG// CHALLENGES Our requirements fromHBase are: • Read performance – fast with low variability • High availability • Operational simplicity • Efficient use of good hardware • Expressive power Bloomberg has been investing in all these aspects of HBase
- 11.
- 12. HBASEATBLOOMBERG// WE’RE NOT THEONLY ONES 12 Google Cloud Bigtable
- 13. HBASEATBLOOMBERG// AIMING HIGHER We canmake things better by working together Let’s be the gold standard
- 14.
- 15.
- 16. HBASEATBLOOMBERG// FURTHER BOLSTER RELIABILITY 16 Greatstrides such as HBASE-10070 but more to do • Improved reconciliation of state between Master, META and ZK • More determinism in Admin/Master operations
- 17. HBASEATBLOOMBERG// BENEFIT FROM MODERNHARDWARE 17 • 32 cores - 256GB RAM – SSD - untapped potential • CPU load max 20% , inadequate throughput • Multi-RS administratively painful • Much better story with memory
- 18. HBASEATBLOOMBERG// IMPROVE MULTI-TENANCY 18 • Mixedworkloads challenging • interactive vs batch • read vs write • different read access patterns • Many solutions in progress • Administrative simplicity is key
- 19. HBASEATBLOOMBERG// SPARK INTEGRATION 19 • Analyticalframeworks need a distributed database • Columnar file format != column database • Integrate with HBase to move towards the universal database
- 20. HBASEATBLOOMBERG// ANALYTICS: EFFICIENCY 20 • Choiceof row and columnar storage engines • Expose primitives for efficiency: • Column pruning • Predicate pushdowns • Data locality
- 21. HBASEATBLOOMBERG// THE FUTURE ISBRIGHT 21 • The state of the “Hadoop Database” union is strong – Increasing adoption – Strong foundation – Great community • Prominent role in the data & analytics platform of the future • Let’s go create the future
- 22.
- 23.
