How we use IRC at Puppet Labs

We’re building out an Operations team at Puppet Labs (we’re hiring!) and one of the aspects of that build-out is the opportunity to play with new ideas and renew our acquaintance with some old ones. And who doesn’t like building green field teams and tools? :)

At the moment we’re frantically busy but we’re also slowly starting to look at tools and to clean up the technical debt we’ve accrued in building a start-up from scratch. Among the areas we’re tackling are communication mechanisms, principally how we all talk as a group.

The last Ops team I worked closely with (but technically wasn’t part of) didn’t use a collaborative chat tool like IRC. There was a strong IM culture (and being a bank - lots of email) but very little collaborative chat across the team. At the time it felt wrong and I tried to push them several times into exploring collaborative chat without a lot of success.

For me I can’t see how an Ops team can be functional without a collaborative chat environment. Email sucks for this - threaded or not it isn’t a real conversation. In turn IM tends to be very point-to-point and most of the group IM solutions suck. Sending alerts, automation and notifications to IM groups is also clumsy and resource intensive. Indeed, having the right place to interact fosters:

• Bonding (largely over memes),
• Knowledge sharing,
• A place to trigger automations,
• It’s a central place to send alerts and notifications,
• It’s often a venue for what passes for humour in Ops (still largely memes).

This is especially true if a team has remote members. It’s one thing to build a culture and work together sitting in the same pod but quite another when you might see your colleagues once a month. In those environments you already have a communications deficit and every tool you can deploy to overcome that deficit is crucial.

Like many Ops teams it was natural for us to fall into IRC as a communications medium. I was initially a bit worried that we’d fallen into IRC because of habit so I did quite a bit of comparing and reviewing of alternatives like Campfire and HipChat. But ultimately IRC still felt right to us:

• It’s a very simple protocol.
• There’s lots of client support, ranging from command line clients like Irssi and Weechat as well as integration into a lot of GUI clients. Additionally it’s easy to run a client in a screen or tmux session .
• It lends itself to automation and tooling with the use of IRC bots.
• We can host it ourselves controlling security and availability.
• It’s open source and free.

There are obviously some challenges with IRC. It’s a text medium and so is subject to the usual potential misunderstandings and has limited ability to convey emotional context. It also requires some knowledge, client setup and has a little bit of a learning curve. And you do need to deploy some infrastructure to run an IRCd server.¹ None of these requirements are particularly onerous though and it’s pretty unlikely we’d ever hire anyone who would struggle to get IRC working.²

In addition to being a place to talk we also use IRC as a home for automation. In most IRC channels there lurks one or more bots that perform useful functions. We have several: an instance of the Gepetto bot that also lives in the #puppet channel too. We’ll decommission that one in our channel shortly. The GitHub bot pops in to tell us about commits. Finally, in keeping with our Puppet theme, we have a freshly created bot named Lampwick.

Lampwick is a Cinch-based bot written in Ruby with a simple plugin system. Being Puppet Labs and open source you can find the source code of the original bot and our fork of it. We currently run it out of Heroku.

We get Lampwick to perform a variety of functions for us including:

• Looking up Zendesk and Redmine tickets for us
• Look up information from a variety of useful sources like Google, Maps, Rubygems, and IMDB (to assist in settling those “Who was in that show about the thing with ‘whatshername’ from that other thing” questions)
• Can send SMS
• Karma (backed with RedisToGo on Heroku)
• And naturally he handles a variety of memes - Magic 8 ball, CoderWall, WhatTheCommit and others.

We’re also going to add a Factoids plugins which is both meme and automation: “James is a doofus” and “Who is on call?” Although we may add a PagerDuty plugin to return the on-call staff member since it’s not the most popular factoid and I sense the potential for SysAdmin abuse resulting in me being all call all of the time.

Future directions for bot plugins include:

• Triggering MCollective and Puppet agent runs
• Reports, querying and running Bacula jobs
• Receiver hook for irccat sessions
• Viewing logs
• Receiver hook for getting Nagios alerts
• More New Relic integration to alert when App performance falters
• Logging the channel (I also run the Puppet Logs site which has about five or six years of history from the #puppet and #puppet-dev IRC channels on Freenode. Searchable too)
• Probably every bloody meme under the sun.

We’re having an automation offsite³ next week to tackle a bunch of these and related automation and I’m excited to see what we get done and what we develop.

P.S. Thanks to everyone who suggested other ideas for useful functions that could lurk in IRC.