The replatforming of the enterprise IT infrastructure utilizing microservices and containers is no small undertaking and is usually provoked by a shifting set of key business drivers. That is precisely the case today. The term “digital transformation” is in the hearts and minds and on the lips of top-level business executives and IT leaders alike. The underlying traditional or legacy infrastructures that have dominated enterprise IT for nearly 30 years simply cannot handle the workloads or power the applications that will drive business decisively forward.
Microservices & Containers
Both microservices and containers are destined to play a major role in this enterprise replatforming, for different reasons. Containers, for their part, hold significant benefits both for developers and the development effort as well as for the organization itself. Understanding this range of benefits is instrumental to securing IT budget for containers going forward.
In this DevOps Chat, Jim Scott, director, enterprise strategy and architecture at MapR, discusses the most promising and compelling tools, technologies and solutions destined to play a major role in this replatforming of the enterprise. Jim highlights how to stay one clear step ahead as IT undertakes what is arguably its most mission-critical task in a generation.
You can also download a great ebook on this topic by Jim at here.
As usual, the streaming audio of our chat is immediately below, with the transcript of our conversation below that.
Audio
Transcript
Jim Scott is Enterprise Strategy & Architecture at MapR Technologies and is very active in the Hadoop community. Jim helped build the Hadoop community in Chicago as cofounder of the Chicago Hadoop Users Group. He has implemented Hadoop at three different companies, supporting a variety of enterprise use cases from managing Points of Interest for mapping applications, to Online Transactional Processing in advertising, as well as full data center monitoring and general data processing. Jim also was the SVP of Information Technology and Operations at SPINS, the leading provider of retail consumer insights, analytics reporting and consulting services for the Natural and Organic Products industry. Additionally, Jim served as Lead Engineer/Architect for Conversant (formerly Dotomi), one of the world’s largest and most diversified digital marketing companies, and also held software architect positions at several companies including Aircell, NAVTEQ, and Dow Chemical. Jim speaks at many industry events around the world on big data technologies and enterprise architecture. When he’s not solving business problems with technology, Jim enjoys cooking, watching-and-quoting movies and spending time with his wife and kids. Jim is on Twitter as @kingmesal.
Alan Shimel: Hey everyone, this is Alan Shimel DevOps.com, and we’re here for another DevOps chat. We’ve got a nice chat lined up today. Lucky enough to be joined by Jim Scott of MapR. Jim, welcome to DevOps Chat.
Jim Scott: Thanks, Alan.
Shimel: Thanks for having – thanks for being with us today, Jim. So Jim – well, let’s start with this. Our audience may or may not be familiar with MapR, but let’s assume for the moment they’re not. Can you give us a little bit of background about MapR?
Scott: Sure. So MapR is a converged data platform. The operable words being “data platform.” So MapR provides a data platform for all data. All types of data, all locations, cloud on premise, and it can run in all of those locations simultaneously. The premise of the platform is to deliver a standard open API architecture on top of our data platform, so that customers can reap the benefits of our platform, but not have to worry about being locked in to the vendor, MapR, for the software that we provide through those open APIs, or even the infrastructure that you choose to run on.
So if you wanna run on your hardware in your own data center, or if you wanna run your software in the cloud or multiple clouds, you can run your software on top of MapR on any of that infrastructure, anywhere you choose, any time.
Shimel: Great. So Jim – and just for our audience information, what’s your role with MapR?
Scott: Oh, so for MapR I’m in charge of enterprise architecture, both internally and externally.
Shimel: Excellent.
Scott: So what that means is internally, I’m basically lining up and showing how we’re using MapR to solve our own problems, and then externally, it basically means I work with our customers a pretty substantial amount, helping them work through how to be successful with a big data platform. So what is the right way to do something. I’ve personally worked in a large number of different industries, and so my experience comes to the table quite often.
Shimel: Got it, got it, got it. We got a little vibration there, but I think it went away. And Jim, as we were talking off-mic before we started today, I was telling you I had covered MapR back in Network World when you guys launched, and I’m gonna guess that was around 2008 or so. Correct? 2009, maybe?
Scott: Yeah, the company went in – or was founded in stealth mode in 2009.
Shimel: Yep, I remember it. So certainly there’s been a lot of water under the bridge in terms of in 2009, cloud itself was quite the rage. It was relatively new, still, and people were still figuring that out. The whole big data market – there wasn’t a market for big data, frankly. Hadoop and these things had just come out.
But Jim, probably one of the biggest innovations or changes in infrastructure over the last two or three years, maybe four, has been the big move to containerization of microservices. I would imagine you’re seeing this at MapR as well.
Scott: Yeah, absolutely. So if we look historically, just say maybe eight, ten years back, message-driven architecture started becoming popular; service-oriented architecture, things of that nature. And that popularity was great – eh, maybe even 15 years back. But the problem was scaling out the messaging platform itself was difficult, it was expensive, and it was really just a pain from an administration perspective, to be able to scale out.
So we’ve seen this transition over the last handful of years, moving more towards a microservices model. And the term – a lot of people like to throw it around loosely and call something a microservice when it’s not, but the less monolithic in nature, the more difficult to handle the large scale.
Which is why the Kafka API and what MapR provides with MapR Streams helps enable the customers, because they have the ability to push through massive volumes at a linearly scaling cost model, so they know exactly where they’re going. So it’s pretty easy to push through something like 1.5 trillion events per day on a platform like MapR with as little as five servers.
And the more micro in nature, the higher the volume you’re going to need to be able to handle. And so those messaging systems of the past – just not there. Just couldn’t handle it. I mean, I remember being excited when we were getting 50,000 to 60,000 messages per second through the system.
Shimel: For good reason, Jim. You’re talking to someone who was excited when the 386 came out. So yeah, I hear you. But Jim, fundamentally what my – and I just had this discussion, because we actually have a little microservices journal, Microservices Neighborhood, within DevOps.com. And I was talking to JP Morgenthal, who’s a CTO at DXE, actually, about this just yesterday.
Fundamentally, fundamentally – and we’re talking about real microservices now, not nonsense, right? But fundamentally, it has such promise to change the way we do development and to change the way our infrastructure runs, right? Especially when you marry it to containers.
Scott: Yep. Yeah, and I think what you’re alluding to, and this is one of the things that I have the biggest problem with when talking with people about microservices, is the fundamentals are that a microservice is really more about the deployment model than anything else.
And without a container to be able to put it in and deploy it, it becomes a very difficult and unwieldy thing, even though it’s so small.
Shimel: Got it. So Jim, you mentioned before you were working on both the internal and external aspects of this. Internally, if you can – let’s not give away any company secrets – but internally, is MapR utilizing your container-based infrastructure with microservices?
Scott: Uh-huh, absolutely. Yeah, matter of fact, we talk about it pretty regularly. So for our own website, we have implemented customer 360 capabilities. We run our software inside of containers, and we run it currently on one cloud, but we have on-premise hardware, and we will be in the not-too-distant future running multicloud and on-premise all simultaneously. And in order to do that you need a technology like a container to get that type of isolation from your infrastructure.
Shimel: Yep. An abstract, if you will; an abstraction – I had an interesting conversation, Jim, with I think it was at _____ recently, or another site, another large organization who had totally moved to a containerized infrastructure, and he was talking about what a profound change it was.
And I asked him about patching containers versus immutable infrastructure, just standing up new ones, replacing the old, rather than actually patching the existing ones. Where are you on that question, if you don’t –
Scott: Well, I love the concept of being able to go back to my base, default image and be able to run it through my entire devops process, continuous integration process. So I can go patch one instance in my dockerfile, I can test the core out, make sure it’s doing what it’s supposed to, and then I can let all of my downstream builds run, pick up the new container, and run them through their integration tests.
Shimel: Excellent. So Jim, I don’t know if everyone in our audience understands this whole idea of eating your own dog food, and what it means. But can you maybe give us an example or two of how things you’ve learned by using them internally at MapR have translated into better service, better product, better results for end users?
So how you’ve taken it out, how you’ve taken certain processes or waves of technology that you use yourself, and translated that into a customer experience.
Scott: Yeah, so there’s a couple that pop to mind, and number one is actually containers. We only introduced support for containers around the beginning of this year.
Shimel: Really?
Scott: But we’ve been using – yeah, it was – I think our press release that we did on it when we officially released support was around February time frame this year. But technically speaking, we have a number of customers who’ve been using containers with MapR for two, three years.
And little secret here: We use it internally, and have been for two, three years as well. So making sure that we had it fully vetted and very well baked from a support infrastructure perspective was really critical, but getting that out there for customers, really important.
Because it solves such a major problem with containers, which is you look at something so simple as security, right. If I am responsible for the web server for my company and you are responsible for the financial systems of your company, or of my – or our company, that software in a container has the ability to run on the same server.
But if they run on the same server and we are utilizing local storage on that server – because you can’t save data inside a container; it blows up. It’s gone when the container dies. So if you save data locally on that server, what happens? Well, suddenly I run these security risks of my web server having access to financial information – probably not a good idea.
Shimel: No.
Scott: So we implemented, within our MapR persistent application client container, the ability to talk to MapR from within the docker container. And because MapR offers NFS support, it’s almost invisible and there’s hardly anything to do. And you give it a ticket from your authentication environment, and you’re good to go.
Whatever’s running inside that container has a user ID, and when it talks to MapR, the data is saved and protected. You don’t have to worry about oh, well, if I’m gonna save data, I’ll save it to the local server. You get rid of that whole problem. It isolates where the data goes, already being protected and highly available, if your server physically dies, just like if your container dies, you don’t care. You don’t have to restore the server to try and get the data off of it.
Shimel: Got it, got it. That’s 100 percent – that’s the beauty of that whole abstraction layer, if you will.
Scott: Exactly.
Shimel: Jim, here’s an interesting question, and I don’t know if you have this info, but I figured I’d ask anyway. We’ve been doing a bunch of surveys about container adoption, and it’s funny, we’ve done several of these surveys now over, let’s say, the last two years, and when we first started, a lot of people had plans of looking at docker and containers.
Other people, a decent amount of people, had containers in their – they were in the testing dev environment kind of thing, not production. Few had them in production. Then recently, like our most recent surveys, let’s say, over the summer, that number shot up. Probably 40, 50 percent of respondents have containers in production.
That may be because it’s DevOps.com, we’re in a devops bubble, and so you get people using this kind of new stuff. But I’m curious, out of the MapR customer base, a gut check if you don’t have the real number, what do you think the percentages are that are using containers in production these days?
Scott: It’s a pretty big number. I don’t have a hard-and-fast number, but if I just go by the customers that I talk with, I feel like it’s gotta be over 50 percent.
Shimel: Really? Wow.
Scott: Yeah. Yeah, we’ve got one extremely advanced customer in Australia, their name is Quantium. And they talk about how they use MapR all the time, and they’ve basically reinvented their entire operating model, and they’ve done it on top of MapR. There’s an enterprise architecture I documented that they talk about periodically; it’s called the zeta architecture.
They basically use an orchestration layer, they use MapR for persistence, and then they use containers to launch and deploy anything anywhere arbitrarily across their cluster of hardware for their customers. And they turned their entire internal platform to an as-a-service model for internal and external customer consumption.
Shimel: Got it, excellent, excellent. We’re hearing stories like this all over now, and the interesting thing is it’s almost wherever you see containers, you’re seeing microservices. But you’re also seeing microservices in places where you don’t necessarily also see containers.
And it reminds me of when I was in school, we used to learn all pasta is macaroni, but not all macaroni is pasta, or something to that effect. But microservices has a place outside of containers, though they seem to just go really well together.
Scott: Yeah, absolutely, and I think when you start looking at the different types of use cases that you would fit into a microservice, this is where you start seeing how people are going to choose to deploy and manage. If you’re looking at event-driven, message-driven, where it’s taking a message in, it’s doing some work and it’s dropping a message back out, versus it’s a restful service endpoint, where it’s a synchronous response.
Yeah, people are choosing to do things in different ways, but nearly – everyone that I talked to, they’re using containers in some fashion.
Shimel: Yep, yep. It’s true in my case too, but like I said, I always figure well, I’m in a bubble, so of course I’m gonna see that. Jim, what’s next for MapR?
Scott: Oh, that’s an in intriguing question. Recently, we announced our 6.0 launch, and we have greatly enhanced a number of features around our database capabilities. I seem to have this conversation a lot with people, is they start looking at big data technologies, and they say well, this is great and all, but I’ve built this software to work with mySQL or Postgresql or Oracle or whatever RDBMS.
So that’s fine, you can containerize those applications if you wanna start changing your infrastructure and how you orchestrate your software, and if it’s important enough to you, you could migrate it over onto a no SQL database like MapRDB document database, or you can just leave it running in a container so you have a more flexible infrastructure.
And the database technology enhancements that we’ve been making are really critical to kind of driving the future of getting people off of those super-expensive to scale RDBMSes that are being used for non-relational purposes.
Shimel: Excellent. Just time frames – when might we see some more on this stuff?
Scott: Well, it’s been announced in the last few weeks, and so we’ll start our 6.0 general availability very soon. I forget the exact date. Then we’ll just continue plugging along, adding more new, fantastic features to help enable the customer base.
Cloud integration with our platform is another really big one, making sure that we can help enable customers to benefit from the concept of cloud neutrality. Cloud providers are never just gonna get together as a consortium and say hey, why don’t we standardize our APIs to the benefit of our customers?
It just doesn’t happen. It’s not gonna happen. Google, Microsoft, and Amazon are never just gonna call each other up and say hey, what do you think about this?
Shimel: I agree. Agreed. Well, not – you know what – well, I was just, I’ve seen a lot of changes over the years, but that’s probably a pretty safe bet, Jim.
Scott: Yeah.
Shimel: Unfortunately. Anyway, Jim, believe it or not, as I told you when we started, the time goes really quick, and we’re about out of time for this version of DevOps Chat. But we’d like to maybe have you back on in the future. We’d love to keep up with what’s happening with MapR, especially as it relates to microservices and containers and some of the new stuff we have going on.
Jim Scott: Yeah, sounds great.
Shimel: Great. But in the meantime, if people wanna get more information about MapR, where can they go?
Scott: Well, I would recommend they go to MapR.com.
Shimel: Okay.
Scott: Or of course, you can always follow MapR on Twitter and/or any of the other social networks that you join and work with. But the website is always my favorite, because there’s a lot of great material there, including the MapR blog, and all of the ebooks, including the “Microservices and Containers” ebook that I wrote is out there, for free.
Shimel: Got it. Got it, got it, got it. All righty, well, we’ll have to maybe get some links to those we’ll put in the show notes of this as well.
Scott: Yeah, that’d be great.
Shimel: Fantastic. Hey, Jim, thanks for being our guest on this episode of DevOps Chat. Look forward to having you back soon and finding out more about MapR. Continued success, and keep us posted about what you guys are doing.
Scott: Thanks, Alan. Appreciate it.
Shimel: All right. Hey, everyone, this is Alan Shimel for DevOps Chat on DevOps.com. Thanks for joining us, and we’ll see you soon on another chat. Take care, everyone. Bye-bye.
