7 Lessons Learned From Designing DEF CON Cloud Village CTF

7 Lessons Learned From Designing DEF CON Cloud Village CTF

Time Stamp: January 10, 2024 8:00 PM
7 Lessons Learned From Designing DEF CON Cloud Village CTF PlatoBlockchain Data Intelligence. Vertical Search. Ai.

Capture the Flag (CTF) events are both fun and educational, providing cybersecurity professionals a way to flex their hacking skills while learning new concepts in a constructive and safe environment. Well-designed CTFs expose individuals and teams to operational challenges, novel attack paths, and creative scenarios that can be later applied in their work both as offensive and defensive security professionals.

But not all CTFs are created equal, and there’s a lot more that goes into designing a successful CTF competition than just coming up with the challenges. Along with the technical design challenges, there are also operational considerations involved with setting up the environment and actually running the competition, creative planning required to set up an engaging game, and factoring in details related to gamifying the challenges, such as tradeoffs in how scoring structure is set up.

“As a designer I want it [the CTF] to be challenging fun. I want to reward people who are clever, who really work at it, and who are persistent,” says Jenko Hwong, principal researcher on Netskope’s Threat Research Labs team and team leader for last year’s DEF CON Cloud Village CTF. “It also has to be practical for us to carry out.”

Fun and practical was the mindset that Hwong brought to the DEF CON CTF, a massive multi-day affair that had over 400 individuals and teams trying their hands at the challenge and a team of 20 working under him to run the event. A veteran researcher and seasoned CTF participant, Hwong had never run a CTF before this event. One of his biggest hopes for his first try at the job was to level up the relevancy and realism of the challenges in the event, which can sometimes be a bugaboo in CTFs today.

“Sometimes in these CTFs you get a really hard challenge but it’s like what’s the point of this? It’ll be a decryption or encryption problem, where the event goes, ‘Here’s something, good luck,’ and then you have to jump through all these hoops that may not be completely divorced from reality but don’t really fit into a bigger storyline,” he says. “So, when I got the call, my thought was ‘let’s jump in, let’s figure out a good story and a good set of challenges that will be fun but also that make sense and maybe relate to the real world of research penetration testing, defensive measures, what’s happening in the real world.'”

As he dove into the project, though, one thing he found especially challenging is how little information there is out there about running CTFs. Most write-ups are from participants who rate an event and explain how they solved challenges, but there’s rarely information offered on best practices in running an event. As a result, he said that he and his team had to do a ton of work was done creating challenges nearly from scratch.

“The community generally shares a ton, so why aren’t we sharing CTF challenges?” he says. “I think we can do better.”

In that spirit of security community sharing, he shares some important lessons that his team picked up along the way so that others in charge of CTF design can learn and understand from the process. His goal is to run the event again and build on what they learned last year. He also hopes others will share their best practices, and even technical details, so that the whole security community can improve the quality of CTFs being offered.

Storytelling is Key

Hwong says that his DEF CON Cloud Village team was very keen on crafting a storyline that was engaging and fun. He says he thought of the story as a movie script with realistic cyber scenarios built in. For the event they chose a theme of ‘Gnomes’ that was fun and funny. but it wasn’t just the storyline writing that was important but also how the technical challenges were planned within the story.

“The goblin and gnomes storyline wrapped around everything but the important thing was coming up reasonable scenarios that you might encounter as a security professional, including attack paths and reasonable defenses you’d encounter,” he says. “The more we can do that as CTF designers, the better it is for learning and the more fun the CTF.”

Take a Software Development Approach

CTF creators should definitely take a software development approach to designing the technical elements of their challenge, Hwong recommends.

“You’ve got to think of design, implementation and testing,” he says, explaining that he and his team learned the hard way how difficult it can be to test challenges in a complex CTF environment that can be manipulated by participants in numerous ways.

“What happened—and I’ll take the blame as the lead creator for not guiding the testing—is we missed the negative testing pass, as well as the viability checks,” he says. “Part of it is we didn’t have enough time to test, so I was continuing to lock down some environments as the challenge was underway so some of the challenges wouldn’t be too easy and there were no loopholes. I think at one point for an hour or two I ended up making something unsolvable at a certain step.”

So, one of the big lessons he learned is that CTF designers need to bring software development rigor to the table that goes all the way through testing and viability work.

Operational Rigor…and a Little Bit of Caffeine

Meticulousness in software development isn’t the only technical capability that needs to come to the table. The crew running a CTF also needs some serious operational rigor as well.

“We had some fabulous people running the servers and the AWS accounts and the Google and Azure accounts and making sure things kept running and that we were monitoring things,” he says. “All of that stuff has to be handled. And if you ignore it, it just could mean things fail, break or you have performance problems.”

One of the operational problems they ran into was that they experienced some collision between participants and challenges, as the team was operating with a constraint in that they couldn’t create a standalone environment for every participant across AWS, Google, and Azure.

“Because it was in the same environment, it helped them on other challenges and if you have a challenge that requires changing the environment then you have people stepping on each other’s toes, changing a shared object,” he said, explaining that he and his team had to reset policies as the CTF rolled forward so participants wouldn’t run into one another.

He and his team are trying to learn from the experience to figure out a practical method—from time, effort, and expense perspective—to give participants a truly isolated environment without making the whole CTF less viable because things break or take forever to execute.

Finally, Hwong says that on the operational front CTF show runners also have to be mindful of the constant communication that they’ll need to facilitate between their team and participants.

“I was on Discord after midnight and I’m like, ‘I’ve got a talk to give in the morning, would you go to sleep?'” joked Hwong, who explained that participants will have questions and they’re going to be pinging organizers for tips and pointers at all hours.

Designing Different Difficulty Levels Is Hard

Getting the difficulty levels of challenges right and creating a fair scoring system may be harder than a newbie CTF organizer may initially think, warned Hwong. He explained that a few of the levels that his team designed as easier were more difficult for participants to complete than they’d anticipated, while some of the more challenging levels were successfully finished by more participants than expected.

Hand-in-hand with the difficulty leveling challenge is figuring out a scoring system that makes sense. After his experience at DEF CON, Hwong is a proponent of doing some kind of Bell Curve scoring system. But he says the problem isn’t as straightforward as instituting a curve. There’s also the issue of normalizing and balancing out the advantage that big CTF teams have in racking up challenge points—an issue that one of the participants provided him feedback about after the event.

“So if your challenges can be divided and done in parallel multiple players, if I’ve got 10 people I will be 10 times is fast. And so there’s an advantage,” he says. “His point was some sort of dynamic scoring levels it a little bit. If there are things that he’s really, really good at, he might be the only one who solves it and he’ll get maximum points. The bell curve will reward him versus scale doesn’t necessarily matter if it’s something in his wheelhouse of expertise in terms of 10 versus one. There’s some debatable stuff here that we have to work through.”

One possibility is making challenges sequential, but the downside of that is it could make the CTF too rigid and linear, and it could create a bottleneck or dependencies that could blow up one or more challenges. Hwong says he’d also love to see more CTFs reward participants on techniques like how stealthily they operate in an environment or dock points if they leave too many footprints and fingerprints, and that’s an area he’d like to explore as he designs future events.

Regardless, though, dynamic scoring is something that could alleviate some of the leveling issues and he and his team are pursing that for the coming year.

Blue Teams Need More Fun CTF Challenges

After working through his first CTF, Hwong also increasingly believes these events don’t do enough to challenge and really engage blue team participants.

“Blue team exercises tend to go like this: ‘We have a misconfigured environment with lots of vulnerabilities. Can you go fix them?'” he says. ” And what they do is they just test whether those configurations are changed or not or whether I can access this public bucket. And as soon as you make it private, we know you fixed it and you get points. It’d be way better to do things on top of that, such as what if you’re compromised, there’s an attacker in your environment, you have to find them and kick them out. So you have an incident going on right now, and as long as the attacker is, they have credentials and as long as they’ll do things, you might be able to detect it. That’s your job as a participant. And until you revoke their access, you don’t solve it and you don’t get maximum points.”

Those kind of scenarios are harder to do but they’re more realistic for defenders and will make CTFs more valuable for them, he says, explaining that is on his radar for next time.

CTFs Need More Fresh and Relevant Components.

Hwong also challenges CTF designers—and himself–to incorporate more fresh exploit and vulnerability information into their challenges. This was one of the things he wished he had more time to dive into in his first go at DEF CON Cloud Village and which he’s resolved to improve for next year.

“This is one of the areas where CTFs can be more of a learning and training tool,” he explains. “We would love to use relevant ideas and exploits fresh from researchers occurring earlier in the year or even presented at DEF CON.”

CTF ‘Building Blocks’ to Improve ‘Reusability’

Finally, one of the biggest lessons Hwong says he learned is that the industry needs to find more ways to create reusable components for CTF just like software developers do for applications. He has dreams of helping to organize an open GitHub repository of small exercises in code that can form the building blocks of building out a CTF.

“You’re still going to have to customize it and add your own twist, but the idea is let’s get the first 60% out of the way so CTF organizers can focus on really novel things. That way nobody is reinventing the wheel,” he says. “And then the remaining 40% can be adding new techniques, scenarios, and storylines.”

Time Stamp:

More from Dark reading