Veracode’s Journey to DevOps: Getting Agile-ish

Pete Chestna By Pete Chestna
August 24, 2018

As I stated in my previous post, in 2012 we started a transition to Agile. Because Veracode was and is always constructively dissatisfied with our current state and we have a culture that embraces learning, we were eager to find a better way. Our internal champion, Tom Hickman, had done this before and he proved himself a great coach and mentor. I will forever be grateful for his guidance.

There was no shortage of things to learn. I became the first scrum master for the team and we jumped headlong into sprint zero and began building and grooming a backlog. Reflecting on that time I think the hardest thing for us was decomposing the stories into reasonably sized chunks. Planning poker was an eye opener as well. Using collective experience and wisdom to size the effort in a story took lots of time in the beginning. There is a natural desire to equate points with days of effort and it takes vigilance to prevent falling into that trap.

Daily standups were the next hurdle. Looking at the horizon of the end of the sprint and giving status that accurately reflects progress toward ‘done’ was hard for everyone. This was compounded by the fact that we started with a definition of done that included quality assurance (QA) testing. Many times the development task would eat most if not all of the sprint and leave little to no time for QA to do their jobs. That led to some frustration for the whole team.

As we worked through the bumps and bruises of transformation, there was a target that I aimed the team toward. More than anything I wanted to accomplish stories in, stories out. In Agile Scrum, a team gathers for a planning session where they select and commit to the work they will do for the upcoming sprint. Sprints are defined by two important things, the work they have committed and the fixed duration during which they will attempt to complete it. In order to take credit for story completion, you must fulfill the acceptance criteria for the story and it must be approved by the product owner at the demo. At that time, we had three different product owners for the stories in our sprints. Time and development surprises always seemed to be against us at the end of the sprint.

One particular sprint though I could sense how close we were and I decided that this was going to be the week that we got it all across the finish line. On the last day of the sprint I raced between the team and the product owners to ensure that all of the demos would be accepted. It wasn’t easy and there was some negotiation and lots of encouragement. In the end the team did it and they were rewarded with a walk for some ice cream and social time. It was some of the sweetest ice cream I’d ever had. I will warn you that it is incredibly difficult to get there every time, especially if your team is aggressive with the amount of work that they commit to, but I believe that it is an important goal. As one of our leaders says, “if you aren’t crashing, you aren’t racing.” Roughly translated to sprint payload it challenges the balance between getting done easily and getting done ‘with a stretch.’

Lest you think we were full of ourselves for our accomplishment, let me tell you that we were more agile-fall than agile in the early days. What I mean is that at sprint end we would deliver our payload to trunk. After a few development sprints we would go ‘pencils up’ on new development and start a hardening sprint to qualify and secure the payload prior to deployment. Our backlog had a bunch of ‘copy me’ stories that would become the template for the hardening sprint. There were many reasons for this sprint. Primary among them was a lack of automated tests for our existing product and the manual deployment process that we were following at the time. Secondarily it took time to run our security analysis and then to address them prior to release. It would be years before we combined regression and security testing closer to the time of development.

We also started to wrestle with Agile at scale and how to manage multiple payloads from multiple Agile teams in the same source repository project. At the time we used branching and code merge days. This phase of our journey was incredibly painful. Instead of reliving it, let me urge you to avoid it at all costs. Trunk-based development is the best way to go under these circumstances. Long-lived branches are evil.

In my next installment I will cover project Purina and our drive to automate and integrate security testing deep into our SDLC.

As Director of Developer Engagement, Pete provides customers with practical advice on how to successfully roll out developer-centric application security programs. Relying on more than 10 years of direct AppSec experience as both a developer and development leader, Pete provides information on best practices amassed from working with Veracode’s 1,000+ customers.

Pete joined Veracode in 2006 as a platform developer and was instrumental in delivering the first version of Veracode’s service to customers. Later, as Director of Platform Engineering, Pete managed the Agile teams responsible for delivering Veracode’s SaaS platform and built the first DevOps team.  Pete also spearheaded Veracode’s initiative to automate the use of Veracode products into the company’s development processes. Using this experience, he has spoken with hundreds of Veracode customers to help them set up similar programs.

Pete has more than 25 years’ experience developing software and has been developing web applications since 1996, including one of the first applications to be delivered through a web interface.