Share
Incident Response: 95% boredom and 5% sheer terror

At one point in my career, I was an incident response (IR) handler for one of the biggest computer networks in the United States, The United States Postal Service. Back then, incident response was a bit of a different beast <Insert going Postal joke here>.

Back then, incident response was a bit of a different beast. The easiest way to describe IR during this era was that of 95% boredom and 5% sheer terror! We spent our days manually looking at logs, writing scripts to automate what we could, and occasionally working on our procedures in the rare case that an “event” was detected. Notice I said “detected” and not “occurred”. We generally didn’t see much action.

I recently had the opportunity to have a video chat with Ryan McGeehan, a founding advisor with HackerOne and former security and incident response director with both Coinbase and Facebook. Ryan spent a good portion of his career taking the 95% boredom of IR and figuring out ways to increase the value of an IR program during those moments of calm. I specifically asked Ryan what makes a good IR program, and after a lot of detailed discussion, a handful of ideas popped to the top of the list.

  • Treat everything like an incident. Desensitize yourself to the process of security incident handling by doing it over and over again. Just like speaking in public, or skydiving, when an incident occurs it will make your heart race and your adrenaline soar. The best public speakers and the top skydivers have hundreds of speeches and jumps under their belts. They have been there and they have done that. While it still may feel exhilarating, to a degree, they learn to manage the panic and enjoy the chase. By treating all vulnerability reports, all minor security events, all inbound attacks, as full-fledged attacks, you will begin to alleviate the stress that comes along with them. While it is possible to take this too far, the point here is that increasing repetition will increase the stamina, understanding, and accuracy of your IR program.
  • Fully load your internal red team. Your red team exercises should be as extreme, severe, and real as possible. I’m not just talking about giving your penetration testing team carte blanche to attack you in whatever way they choose, I’m talking about giving them everything they need, and then some, to take you company to the brink of hell. I’ve always said that hacking and penetration testing are asymmetric in nature. The attacker has the luxury of time while the penetration testing team is limited in scope and resources when looking for methods of exploitation. Solve this asymmetry by fully loading your red team with as much capability, resources, and time as you can give them. At the end of the day, when there is no smoke, the best IR teams will find a way to make their own fire… and then respond to them.
  • Make your IR process disciplined, yet agile. Think of your incident response processes as agile IR. You can view using agile techniques as the “scientific method” for IR. From the moment an incident is reported, all the way through the completion of the incident post-mortem exercise, the best IR teams are consistent with their meeting cadence and process. Treat your IR meetings like agile scrum meetings. While the meeting cadence may vary, sometimes daily, sometimes weekly, and sometimes even hourly, the meeting process should hold steady. Step one: Add any indicators of compromise to your incident log. Step two: Update your incident timeline. Step three: Answer any questions you had from previous agile IR meetings. Step four: Ask all of your new questions for this incident. RINSE AND REPEAT! If you do this consistently, everyone will know what to think, what to feel, and how to progress to completion for any incident.
DON’T PANIC!
  • Don’t PANIC!Last but not least, do not panic! It’s really tempting to throw your hands up in the air and shout “Holy Crap.. THIS IS INSANE! <mic drop>”. But that won’t help anyone. One trick is to make sure the security team wasn’t relied on to make everything secure to begin with. I know that sounds ironic, but it’s an improved way of thinking about security. There’s no “scale” team, everyone designs for scale. Why is there a security team? They certainly can’t secure it all. Stay calm and build a company culture of security to allow your team to execute under the pressure of a legitimate attack.

My early 2000s experience with IR isn’t the norm today. As a matter of fact, with the expert work of people like Ryan, and others, incident response has gone from a little side project to a mandatory component of a successful enterprise security program. Spending your money on the detection, and blocking, of as many attacks as possible makes some sense, but if you leave out formalizing the response mechanism you are missing out on a huge piece of the security arena. Remember, with the right preparation and correct views into IR, you can change the IR ratio from 95% boredom and 5% sheer terror to 50% preparation and 50% actually catching the bad guys.


Signal Sciences’ industry first Next Generation Web Application Firewall was built in response to our frustrations of trying to use legacy WAFs while enabling business initiatives like DevOps and cloud adoption. The Signal Sciences NGWAF works seamlessly across cloud, physical, and containerized infrastructure, providing security prioritization based on where your applications are targeted, and blocking attacks without breaking production traffic.