The Technical Operations (TechOps) department of GDS came together and took part in a department-wide ‘game day’. I know the first question I asked is probably the same question you’re asking: What is a ‘game day’?” When I first heard the name I thought this is going to be one of those epic days out when we go zip-wiring or hike across the O2. All these things would have been exhilarating but could not compare to what we actually did.
What is a game day?
A game day is where a multidisciplinary digital team gets together to solve a simulated incident.
So how did it all begin?
Every quarter the TechOps department of GDS holds an away day, where the department gets together and looks back on the achievements of the last 3 months and forward to what’s coming up next. Feedback from previous away days said that the day was full of senior management talking at them and attendees wanted something more interactive. So it was put to all to come up with an alternative. You may think there were many ideas flying
from the ether, but actually there was only one. It only takes one spark to create a fire. Don’t worry though - this was a simulated fire, so no trees were burned in the process.
The idea was to hold a game day across all of TechOps. The aim of this game day was to alleviate the silos that had developed across the department and increase communication between individuals on different teams. It also aimed to be interactive and engaging, encouraging everyone to participate.
Emma Pearce and I took the helm to make this TechOps-wide game day a reality. We punted for technical volunteers, who would have to come up with the fake incident and build any necessary technical components to form a working group.
When the working group became operational, the first thing that we had to work out was what the incident would be that the multidisciplinary teams would have to work on. This in itself was a task, as we found that it was akin to Alice and the Rabbit hole. The engineers and cyber security analysts in our ideas session were coming up with many weird and wonderful ways to make things crash and explode, in a digital way. However, as the delivery managers in the room we had to refocus everyone on what was manageable and achievable that would fit the objectives of the day.
It was therefore decided that we would create one incident with multiple layers, which each team could simultaneously work through.
What was the scenario?
A multidisciplinary service team has inherited a legacy service with no documentation on how it works, built with open source code. There is no infrastructure-as-code set up for the application. The only record of how the machine is built is the cloud-init (user-data).
Now the real fun began - although participants may not have agreed - as we started to orchestrate the chaos. Like every good war game this generated a fight or flight reaction from the participants. Some rose to the challenge and some sunk under the pressure. But, like with any game day, the different layers of the incident tested our resilience and saw us pulling together to get through it.
How did the players find it?
There was a mixed reaction from the participants - mainly positive but all constructive. Here are some quotes from the feedback we received, from some of those who took part:
I thought it was great! More please!
I would prefer if it lasted longer, or if there was more time before the incidents started.
Some of our team members do not like the live scoring as it adds stress to our team.
Hats off to the organisers for the amount of preparation and their responsiveness to events while the game was running - it was ambitious and maybe not everything worked exactly as planned, but they did an excellent job and deserve lots of credit.
Did we achieve our objectives?
If nothing else, the one thing this game day sparked was a reaction, and compared to the absence of one to previous away days, I’d say we did something right. If you remember, we set out to:
- Alleviate the silos that had developed across the department from there being little to no communication between individuals on different teams.
- Have an interactive and engaging away day where everyone could participate.
After the game day we sent around a poll and 95.2% of people said they got a chance to work with people they don’t normally speak to. 82.4% of people said they found this away day more engaging than the last one. All in all I’d say we hit the mark.
It was no small feat but definitely a great day. I’d like to say a massive thank you to all the organisers and players: thank you!