Uploaded byHBaseCon
Apache HBase in the Enterprise Data Hub at Cerner
The document discusses the integration of HBase within an enterprise data hub, focusing on its role in supporting various business solutions while ensuring secure and efficient data management. It highlights the importance of memory management, tenant isolation, resource allocation, and security measures, such as Kerberos authentication, for optimal performance and user coexistence. The document also outlines future improvements for namespace management and quota support in HBase.
More Related Content
Similar to Apache HBase in the Enterprise Data Hub at Cerner
Apache HBase in the Enterprise Data Hub at Cerner
- 1. HBase in the Enterprise Data Hub It’s all about co-‐exis0ng peacefully! Swarnim Kulkarni May 24th 2016
- 2.
- 3. innova8on noun in·∙no·∙va·∙0on ˌi-‐nə-‐ˈvā-‐shən 1. the introduc0on of something new 2. a new idea, method or device
- 5.
- 6.
- 7.
- 8.
- 9. HBase sits at the core of a lot of these solu8ons!
- 10. Each solu8on built with specific business problem in mind
- 11. CHART SEARCH SEPSIS MILLENNIUM + POP HEALTH
- 12. CHART SEARCH SEPSIS MILLENNIUM + POP HEALTH
- 13. Prevents assets from being shared High investment High maintenance costs High barrier of entry for innova8on
- 14. Enter The Data Hub
- 16.
- 17.
- 18.
- 20.
- 21.
- 30. Innova8ons Source of data Raw data
- 31. Data Hub Owner Data Hub Consumers Workers
- 32. “An Enterprise Data Hub is a centralized loca0on to store all data, for as long as needed, in its original fidelity; with flexibility to run variety of enterprise workloads including batch processing and interac0ve SQL -‐ with robust security, audi0ng and management.”
- 33. “An Enterprise Data Hub is a centralized loca-on to store all data, for as long as needed, in its original fidelity; with flexibility to run variety of enterprise workloads including batch processing and interac0ve SQL -‐ with robust security, audi0ng and management.”
- 34. “An Enterprise Data Hub is a centralized loca-on to store all data, for as long as needed, in its original fidelity; with flexibility to run variety of enterprise workloads including batch processing and interac0ve SQL -‐ with robust security, audi0ng and management.”
- 35. “An Enterprise Data Hub is a centralized loca-on to store all data, for as long as needed, in its original fidelity; with flexibility to run variety of enterprise workloads including batch processing and interac0ve SQL -‐ with robust security, audi0ng and management.”
- 36. “An Enterprise Data Hub is a centralized loca-on to store all data, for as long as needed, in its original fidelity; with flexibility to run variety of enterprise workloads including batch processing and interac0ve SQL -‐ with robust security, audi-ng and management.”
- 37. Mul8-‐tenant Secure and Compliant Ac8ve archive of all data Low barrier of entry Hadoop as a Service Data Hub is….
- 38. So why provide HBase as a service?
- 39. Maximize cluster u8liza8on Minimum resource guarantee sharing Low barrier of entry HBase as a Service Knowledge Data Data
- 40.
- 41. Support mul8-‐tenant environment for mul8ple users Isolated deployment Set quota per consumer ACLs for project level administra8on Appropriate security for user authen8ca8on
- 42.
- 44. Smaller HBase clusters part of the bigger “hub” Scale as per need Easier management
- 45.
- 46. Making sure that you have sufficient resources before you start the job!
- 47. CGroups • Control Groups • Police and limit CPU, Disk I/O and Memory usage • Resource guarantee by sta8c par88oning • Very useful in case of conten8on
- 48. • Value for resource-‐limits driven by individual tenants depending on the workload • Defaults to 70-‐30 usage for memory and CPU between (NodeManager + Datanode) and RegionServer • Higher memory on RS (at least 4 GB) helps avoid swapping and happier GC CPU CPU CPU MEMORY MEMORY MEMORY SEPSIS M+ POP HEALTH
- 49. What we learned • Recommend tenants to give atleast 4 GB of memory to the regionserver for smoother opera8on • Coopera8ve memory (ex: via JVM heaps, max. container sizes) works bejer than cgroup limits • Disable swapping on HBase nodes • Disable HDFS load balancer • CMS GC performs way bejer • Limit number of containers running on nodes to maximize performance
- 50. Property Value yarn.nodemanager.container-‐ executor.class org.apache.hadoop.yarn.server.node manager.LinuxContainerExecutor yarn.nodemanager.linux-‐container-‐ executor.group hadoop yarn.nodemanager.linux-‐container-‐ executor.resources-‐handler.class org.apache.hadoop.yarn.server.node manager.u8l.CgroupsLCEResourcesHa ndler yarn.nodemanager.linux-‐container-‐ executor.cgroups.hierarchy /yarn yarn.nodemanager.linux-‐container-‐ executor.cgroups.mount true hjps://hadoop.apache.org/docs/current/hadoop-‐yarn/hadoop-‐yarn-‐site/NodeManagerCgroups.html
- 51. Request queues • Priori8ze variety of workflows that need access to HBase • Important to meet SLAs for tenants • “FIFO” vs “Deadline” queue type • “Deadline”(default) proved to work prejy well for most of the cases • Cannot be set per tenant but has to be set per cluster – support coming soon
- 52. Quotas • Makes sure that no single tenant abuses the system • Peaceful coexistence • Promotes a pay-‐per-‐use model. Could buy a higher throjle limit by contribu8ng more number of nodes • Usually we set throjle on a per namespace basis but could also set on the per table/user basis if needed
- 53.
- 54. Namespace • Logical grouping of HBase tables • Analogous to databases • Provides tenants with individual space to operate on • Tied to the AD group used when onboarding • Could also apply quotas(max regions/tables) per namespace(HBASE-‐8410) but not using that feature for now
- 55.
- 56. Authen8ca8on • Cluster secured by Kerberos • Disallow impersona8on • Require kinit to first authen8cate with the KDC before accessing the cluster
- 57. ACLs • Provides the authoriza8on piece • Set per namespace and 8ed to the AD groups • Required proper8es
- 58. ACLs <ac0on> -‐ Determines the type of ac8on – grant or revoke <en0ty> -‐ Determines en8ty to grant access to – user or groups <level> -‐ Determines the access level – RWXCA <scope> -‐ Determines the scope for access – namespace, table, column family or Cell Must be a super user to run these commands (Determined by hbase.supersuser)
- 59.
- 60. ~33% of all US healthcare data
- 61. Millennium + 4 CDH 5.5.2 clusters (soon to be 5) 1245 tables 113,982 total regions 673 regionservers 895 TB of data! (unreplicated)
- 62. Popula8on Health 900 tables 2 million requests/sec/day 780 regionservers 115,000 regions 700 TB of data! (unreplicated)
- 63. Lots of common data…. • Reference Data • Provider informa8on • Insurance informa8on • SNOMED/ ICD9 / ICD10 data • Ac8vity Data • Visits • Procedures • Vitals
- 64. Crawler M I L L E N N I U M Collector HBase Cluster Crawler Collector HBase Cluster M+ Pop. H Makes sense to be hosted in the Data hub instead!
- 65. Onboarding – Capacity planning M+ Popula8on Health
- 66. Onboarding -‐ Deployment M+ Pop. Health DATA HUB Tenants could choose to modify the cgroup configura8on depending on expected workloads or just stay with defaults
- 67. Onboarding -‐ Isola8on • Create AD groups • poph_users, pop_admins • mplus_users, mplus_admins • Create namespaces (as super user) hbase(main):001:0> create_namespace ’mplus' 0 row(s) in 0.5650 seconds hbase(main):001:0> create_namespace ’poph' 0 row(s) in 0.7262 seconds
- 68. Onboarding -‐ Quotas • Setup quotas hbase(main):001:0> set_quota TYPE => THROTTLE, NAMESPACE => ’mplus', LIMIT => ’10000 req/sec' 0 row(s) in 0.7255 seconds hbase(main):001:0> set_quota TYPE => THROTTLE, NAMESPACE => ’poph', LIMIT => ’1500 req/sec' 0 row(s) in 0.5677 seconds
- 69. Onboarding -‐ Security • Kerberos secured. • Require kinit for regular users to access the cluster • Deploy keytabs for service users • Setup ACLs hbase(main):001:0> grant '@poph_users', 'RWCX', ’@poph’ 0 row(s) in 0.3250 seconds hbase(main):001:0> grant '@poph_admins', 'RWACX', ’@poph’ 0 row(s) in 0.4332 seconds
- 70.
- 71. What we can do bejer • Namespace quota support (HBASE-‐8410) • Limit tables/regions per namespace • Region Server Groups (HBASE-‐6721) • Pin namespace/tables to subset of regionservers • Advanced namespace security (HBASE-‐9206) • Higher flexibility to admins and tenants for namespace management
- 72.
