Author Archives: patrick.sullivan

WHERE WE’RE GOING, WE DON’T NEED ROADS… BUT WE WILL NEED FLAGS

In the previous blog post I introduced our stripe-ctf-2-vm, a self-contained capture the flag puzzle ladder in one vm. In this post, I’d like to talk about how we used the vm to introduce the security mindset to our developers here at Bazaarvoice.

One of the tenets in R&D is to responsibly “fail fast, win fast” with data-driven decisions. It was time for a grand experiment. We put on our lab coats and called for a dozen volunteers for a test training session. The format of this session ran like this:

  • Each participant was a lone competitor.
  • Discussion was encouraged to promote collaboration.
  • Each puzzle was timeboxed based on the feel of the room. 
  • Each puzzle/timebox would conclude with a discussion of the solution.  

sshot-104

What We Learned, Part 1

Gamifying learning is hard. There’s a whole field dedicated to this for a reason. We felt like we hit the broad strokes by providing: an increasingly difficult ladder of challenges, competition, social interaction (almost, see below), and timely feedback. The experiment led us to some very important tweaks…

  • Lone competitors don’t collaborate effectively, for obvious reasons. Attendees should form small teams of three or four for a better social element. Consider mixing technical skill sets and/or functional teams. That really helps.
  • There’s a balance to be had between freedom to play with a puzzle and frustration. It’s a good idea to track the time closely and cut to the solution before tables are flipped.
  • You also need to track the time so you can make an educated guess on how long future training sessions should be.
  • Proctoring and encouragement is needed for those new to CTF or security puzzles. This is easier to do when small teams are formed.
  • Prizes rewarded per level don’t motivate very much.

Taking It to the Big Leagues

Armed with what we learned and bolstered by testimonials from some of the attendees, we proposed to management a way to take this further. Now, BV R&D management is very pro-learning and our VP takes security very seriously. We didn’t have to sell the Why very much, just the How. Since this education was seen as vital to the growth of our engineering effort, we made the case that all one hundred and twenty plus of our engineers needed to train. We provided a plan for about nine sessions with about twenty engineers each. Once we had management buy-in, we did a two minute elevator-pitch to all of engineering so they had context for the meeting invites that followed.

If you don’t have rapport with your management about the value of secure code, you have your work cut out for you. We suggest you leverage your own talent to locate a few security flaws either through code review, pen testing, or fuzzing tools if you’ve got the skills. The likely low-hanging fruit for a web app exploit are unsanitized inputs, cross-site scripting, and SQL injection. Put together a presentation that educates on the rising number of web app breaches, the cost of fixing flaws in production, and the fact that you just found flaws in production. Paying for external pen testing or security training, or security tools can be expensive. Use what leverage you get from the presentation to instigate a bottom-up approach like a book club or training like this CTF session to cultivate security mavens who can advise the engineers around them.

tumblr_l9ofytzjp61qb7unno1_500

The Pattern for a Session

Once you have roomful of engineers ready get their security on, what do you do? We divided them up into small teams and tried to get a mix between front-end, back-end, and QA skill sets when possible. The next thirty or so minutes were spent on helping the participants setup their VMs and log into them via SSH. Most of the issues you will run into will likely stem from mix-ups with localhost VBox adapters or trying to run and log into the VM directly from the VBox UI rather than via their terminal and shell script. When your engineers are logged into the VM, you’re ready to run through the introductory slides.

As soon as fingers are poised to take on the puzzles, give everyone the password and present the staging slide for Puzzle 0 (0, because we are programmers). Set up the puzzle and let the teams go to town on it. It’s a good idea to remind everyone to be vocal about looting the puzzle. And make sure they know they can look at the source code in the VM or in the github repo (no, the code you can see in the levels/0 directory is not the running instance of it so don’t modify it and expect to see the changes). A few minutes later, show the hint slide. Feel out your audience though. Easier puzzles may not need a hint, harder puzzles may need more breathing room before or after the hint is shown.

After you have a winner, or better yet, wait until you have two victorious teams (you don’t want to move too fast), show the solution and remediation slides and talk through ways the vulnerability could have been avoided. Any examples specific to your stack or technology you can use to bolster the remediation discussion will strengthen the discussion enormously. Once all the teams have unlocked the next level, repeat until you run out of time or until you solve all the puzzles. The latter means you are a bunch of badasses. Srsly!

maxresdefault

Go as far as you can, and reward the team that conquered the most puzzles with something highly coveted: Pokemon figures. Or money. Whichever.

Your mileage will very of course, but we found the following timing worked for BV R&D across our three hour training sessions:

ctf-timings

What We Learned, Part 2

Here’s what we learned from conducting nine sessions:

  • Scheduling 120+ people for a three hour training session is hard. Set aside admin time to handle rescheduling requests and plan for a make-up session.
  • Generate a VM per session so the loot is different each time.
  • For our engineers, a three hour session allowed the class to work through the first 5 or 6 puzzles as a group which made for a good introduction to security vulnerabilities.
  • Send out instructions and links to your generated VMs in the meeting invite.
  • Even so, plan on spending the first thirty to forty minutes setting up VMs.
  • Don’t send out the ctf user password in the invite; some clever, motivated individual will work through all the problems before the session. 🙂  If they do that, they should proctor the session with you.
  • If your company uses HipChat, Slack, or similar, set up an invite-only room per session for that session’s attendees only. It helps to have a means of cut/pasting the solutions to the group.
  • Don’t forget to work the room to give encouragement and redirect rabbit holing. Each team is different and some prefer to be left alone. Learn to feel that and respect it.
  • Some engineers will get hung up on the implementation details of the puzzles: “I don’t need to learn bloody PHP.” Try to impress upon them the larger vulnerability exercised by the puzzle. Try leading them through the thought exercise of how the security flaw (and not the language-specific flaw) may be at work in your company’s code today.
  • Some teams will race ahead after unlocking the next level. Most of the time that will even out as the CTF continues. Occasionally you have an ace security-minded team. If they continually blaze ahead, you may need to get their buy-in to take them out of direct competition with the other teams for morale’s sake. Perhaps they can help proctor the others?
  • Use something simple like a GoogleDoc form to take a quick poll of participants after each session. You will get invaluable feedback on how to better customize this to your organization.

We hope you find this unique way of introducing the secure coding mindset to your engineering organization useful and fun. It is just the tip of the iceberg. Once your engineers are thinking this way, you will need to develop further plans to encourage more education on the common secure coding patterns themselves. Consider periodic secure coding code reviews or pair programming with the experienced security-minded folks on your team. Forming teams to compete in other CTF competitions and internal security bug bounties might also help. Please don’t hesitate to contact us if you want tips on how to implement this for yourself or if you just want to let us know how it worked for you.

Ah, But Do You Have a Flag?

Hey you there, did you know that forty percent of all data breaches are due to web application vulnerabilities? That means the very software your team is building is likely to be the vector to getting your data pwnd. Still feeling skeptical? You should google Heartland’s 2008 breach, eBay’s XSS vulnerability, or Time Warner’s password leak. I’ll wait.

Done? Pretty scary, isn’t it? 

Great, But How Do You Get Your Developers Thinking About Security?

The discipline of bullet-proofing your code against application vulnerabilities is called Secure Coding. You want this fancy secure coding to up your AppSec game, but what if your R&D organization lacks the skills? You hired smart people, they can learn it, but they need to want to.  They need to feel it. So how do you get your team stoked on security awareness (besides telling them to stop writing their passwords on post-it notes)?

The first thing you do is put together a rip-roaring slide deck with the top ten security flaws and a snazzy background and get them to read the heck out of it.  Developers love slide decks.

Hmm, That didn’t work.

If only there were a better way, more engaging way.  And there is. Did you learn to code just by reading about Java? No way. You started working on coding examples to get the hang of it, right?  Maybe you even got your code katas or koans on so you could motivate yourself. Why not do the same to cultivate some security awareness love?

0p7ejqm

AppSec enthusiasts commonly compete in Capture The Flag contests. No, not this. Not this either. There are a couple of CTF formats out there, but the Jeopardy format is the one that best suits the needs of introductory training. This format is made up of a ladder of increasingly difficult puzzles. The ladder works like this:

  • Look at the puzzle, in this case a flawed web application. Since we’re interested in secure coding, look at the source code the app.
  • Throw what you can against it.
  • If you succeed in exploiting a flaw in the app, you should get it to cough up the key to the next level. That’s called the loot.
  • Use said loot to unlock the next level.
  • Lather. Rinse. Repeat until you are the first one to loot the final level.
  • Stand on table and celebrate your victory.

You might be thinking, “This is great advice but how do I get me one of those CTF contests?” We thought the same thing. We didn’t have time to wait for someone else to put together a competition, and we wanted to make inroads on secure coding training in a more controlled environment. What to do?

What We Did

Some of us had competed in the Stripe 2.0 CTF like, 37.5 computer years ago (that’s roughly 4 years ago in people years). Fortunately, the good people at Stripe open-sourced those very same web app puzzles. Yea! But they had languished untouched in the backwaters of github. Boo!

We needed…

moar

After some studious digital archeology in the form of ancient version management, we resuscitated the puzzles. Once we had the puzzles in hand, we used veewee to roll a VirtualBox (VBox) compatible VM with some scripting magic to auto-generate the loot values. In this VM, each puzzle was set up to run sandboxed away from the casual user, but still gave them access to the source code.

How Can You Do the Same?

yougetavm

If you’ve read this far, you might be ready to introduce some CTF-based training to your organization. You might still be thinking, “This is great and all, but how do I get me one of those CTFs?”  Scroll no further, true believer.  We have open-sourced all the material you need to conduct your own CTF training right here in this very github:

https://github.com/bazaarvoice/stripe-ctf-2-vm

The instructions to roll the VMs can be found here.  The slide deck needed for the training sessions can be found here. It’s like today is your birthday!

In the next post, I will explain how we developed an introductory secure coding training session around this vm and provide advice on how you can do it too. Now, go get that flag!