DevOps is about knocking down barriers between developers and operations teams, and in doing so reducing organizational frictions wherever possible. And that’s certainly the spirit behind ChatOps – the term for the hyper-collaborative way of running DevOps, including operating aspects of systems and infrastructure, through online chat.
Proponents of ChatOps say it streamlines communication in a centralized and powerful way. Born within Github, ChatOps evolved from its open source chat bot, Hubot, which has changed the way the web-based Git repository runs operations through the automation of deployment, graphing, monitoring, provisioning, and even mitigating security events.
Think of Hubot, a Node.js app that is commonly run on Heroku, as a robot that helps keep everyone informed in a centralized place and in real-time, or in a near real-time, way. There are many benefits to ChatOps, says David Mortman, David Mortman, contributing analyst at Securosis, including the integration of ops activities with notifications into one place, making it easier for everyone to know what’s going on across the enterprise, as well as everyone throughout the organization having the ability to see error messages on events as they occur, he says.
Github, as a highly distributed organization, with its several hundred employees spread across 103 cities, can use all of the communication efficiencies it can muster. As Mark Imbriaco, Ops engineer at Github, in his presentation ChatOps, Augmented Reality for Ops, uses Github, email, internal apps, other web systems that they’ve built to help teams communicate asynchronously – and yet they needed an option for the real-time or “semi”-real time communication. “I say semi-real time because, when we talk about chat, the social contract at GitHub around chat doesn’t require you to be immediately responsive to everything you see. Chat is still asynchronous, but it’s more synchronous than GitHub or email or other sorts of notification mechanisms,” said Imbriaco.
Without enhancement, Hubot doesn’t do a whole lot right away. But when added to chat with one of the dozen or so connectors that include access to Google Talk, IRC, Skype, and Hubot monitors for events that come from within chat or from one’s systems and applications. “The magic happens when you build additional functionality into him to automate, well, everything,” Imbriaco said.
Hubot will do certain things, such as notify people not present in a chatroom that they were mentioned, phrases can be registered so that notifications to those interested in certain events will be notified. Github has also plugged Hubot into the GitHub service itself. Today, when anyone creates an issue or pull request in GitHub, according to Imbriaco, a link and definition of the event is sent to chat. While Github already has ways to send notifications, by entering events real-time into the chat stream, others can jump in and comment, help, or otherwise advise on events and interact as they’re occurring.
As for the adoption of ChatOps, Github isn’t the only one: but it’s not yet widespread among DevOps practicing organizations, either, based on our reaching out to numerous organizations. Nor, is the concept especially well known.
That’s not slowing Owen Zanzal a developer at systems management provider VividCortex down from adopting ChatOps. Zanzal says in his blog post, ChatOps at VividCortex, that, despite ChatOps being off to a slow rise, “ChatOps has really revolutionized how we operate,” he wrote.
The first Hubot mission at VividCortex was to help improve how code is deployed. The problem Zanzal highlights is that encryption keys and new developers apprehension around code deployment, was slowing down production.
To streamline deployment, Zanzal moved the execution of Capistrano commands to a Jenkins (a continuous integration server), and created Hubot commands so that jobs could be created using a Jenkins exposed REST API. When done, developers needed only run a chatroom command, such as “hubot stage webapp” or to deploy “hubot release webapp,” Zanzal explained. “Now, no one is blocked by the gatekeeper from pushing code into staging or production, and every developer can push code when they are ready. The unfamiliar Capistrano commands were simplified into much easier to understand Hubot commands,” he wrote.
VividCortex is also using ChatOps to fetch logs and more quickly remediate issues as they arise, Zanal wrote.
Chat-as-Ops
GitHub has also pushed ChatOps much further than simply as a place to stage and deploy code. It’s using ChatOps as a way to help actually run ops. For instance, Imbriaco spoke to how Hubot and ChatOps is used for efforts such as rebalancing shards across its file servers. “Honestly, it took me a little while to get used to some of these. We have some [Hubot actions] that do things like rebalance shards across our file server. And that’s horrifying to me, because the consequences of failure are so bad. But the reality is, I’m either going to run it from a chat room, or I’m going to run a shell script from a terminal. If I ran it here, at least everyone saw it. So at the end of the day, this is repeatable, it’s tested, it’s commonly used,” Imbriaco said. “And if it stopped working we would notice,” he said.
And one time when Github was fending off a distributed denial of service attack, Imbriaco needed only enter the name of their service provider to direct traffic and the attack was successfully mitigated.
Moving to ChatOps isn’t without risk, however, Imbriaco explained. A lessoned learned when it became a little too easy to deploy to Amazon Web Services. “We went from zero to way too much spend on Amazon in about two weeks, when I turned up the ability for us to provision EC2 hosts from Chat,” he said. But that issue was resolved by giving users information at the time of provisioning, such as including metadata into the instance that details who created it, as well as informing users how much it’s likely to cost,” Imbriaco said.
Despite the efficiencies of ChatOps, don’t expect a widespread embracing of ChatOps, but more likely a slow adoption. “GitHub is a flying unicorn. They’re like Netflix,” says Mortman. “People will pick up some of what they are doing but it’s too advanced for most,” he says.
