We iterate to learn

Posted on June 4, 2020 by Kenny Shen

Some months ago a friend shared Seth Coster’s talk Crashlands: Design by Chaos with me, which I really enjoyed (and re-watched a few times). A big takeaway of the talk for me was how such a purely iterative process could drive realizations of ideas and learning better than laying down complicated roadmaps and plans. For one, it naturally induces an acceptance of what good enough looks like when one set out to build something, and it would be not really be chaos per se, but driven by a directional iterative roadmap.

The talk also mentions the topic of crunch - that is when schedule for an ongoing project has started to slip and everyone is burning the late night candles in a bid to try and complete in time. Unfortunately, “in time” usually also refers to some arbitrary deadline that has no actual justification. (I’m reminded of Jatin Shridhar’s blog post on who are you trying to impress with your deadlines?)

Are iterations really the silver bullet to better delivery on software projects? Are crunchtimes avoidable? The following is a brain dump of sorts as I reflect on what drives better delivery in development teams and processes.

Beware of drift

In software projects, it’s not uncommon to hear of plans going awry, deadlines missed and features taking a disproportionate timeline to delivery and more ominously, features yielding very low returns after all the trouble to get them to customers. Contributing factors I believe that lends to this type of scenarios happening are overplanning and not understanding how to drive an appropriate unit of value for the end customer.

Let’s consider startups. Startups are often thrown to the deep end from day zero: figuring out what customers want, prioritizing what to build, and then deciding what a reasonable iteration looks like. This goes to some prototyping stage, testing and then a planned delivery to the end user, gauge feedback and reception, and iterate or move to the next piece in the pipeline.

In a mostly ideal type scenario, we might start off some hypothesis that customer would pay for X, and after a prototype, we would test this out with a select group of users quickly, and decide what changes to iterate on next - in a week or two. Speed matters, because short delivery cycles actually help with cost management and acts as a natural defense to scope creep. If developers run into blockers, it’s usually along the lines of a day to two, and these can actually help inform the product owner on any miscalculations or planning. All in all, the lower the resolution, the easier it usually is to realign or course correct. Think about trying to draw a 16x16 favicon.

In the real world however, what often rears its ugly head is the concept of drift. In a lot of scenarios where I’ve observed this, decrementalism is culpable. We start off with an ill-informed picture of who our customers are, what is valuable to them and what should be worked on. This quickly gets delegated downstream to product design, sprint planning and long delivery cycles. By the time we realize a project and put it in front of the customers, we’re often shocked to hear this isn’t what they want. Big lofty plans, little room for corrective realignment together with the absence of customer/market understanding brings this “compounding interest” (albeit, the negative variety) to bear on the project over time.

And often, you hear developers working under such conditions burn out and feel demoralized, as the product owners doubles down on the ill-fated product. In learning what separates successful software deliveries from the failed ones over the years, I’ve learnt this: minimizing the drift across the whole process of planning, design, development and deployment is not an optionality. Small iterations paired with knowledge of good enough coupled with speed and brutal honesty and transparency can help avoid drifts into failures.

Let developers iterate safely

Here’s a IRL incident that happened recently.

Some weeks ago, I was summoned to a late night Zoom meeting by a friend who was experiencing a massive spike in their website traffic and the servers were coughing up and dying. As you can imagine, this wasn’t a great spot to be at 11pm. There were many folks on that call - including the marketing, sales, operations teams on top of engineering. When I chatted with one of the developers later on his thoughts post-mortem, he cited badly written code, lack of proper scalable infrastructure planning among some of the driving factors leading to the crash that night. Were these issues brought up before? Yes, he said. What was the product manager’s response when they were brought up? The reply - Refactoring and improving the codebase didn’t add business value.

My response to the developer was this: An engineer provides business value when he does his job well - keeping features bug free, servers up and available, relieving others to execute their responsibilities without worry.

Having a whole group of people up at 11pm doesn’t add business value, though in startups there’s often this misguided sense of heroism that sets up a vicious cycle. The same processes left unchange repeat the same outcomes; in some warped situations, the development teams that often creates the problems ends up being celebrated when they stay up late to fix the problems. For teams to iterate successfully, psychological safety and being able to give feedback (and have the feedback taken seriously) are important aspects.

Doing is not iteration

It’s easy to confuse action with iteration. Someone pointed out to me that learning always accompanies an iteration. If you were trying to sell a SaaS product, you would maybe try to reach out to your prospect the first time via a certain proposal and channel. After that, you might learn the customer was confused, or maybe the channel you chose wasn’t effective. So you assimilate the lesson, and you change up your game. Rinse and repeat. After some time, you would end with a new set of methodologies with how you approach selling the product to a set of customers.

Now imagine the above scenario but after meeting the first customer you didn’t change anything. You continue to bring the exact same proposal, same pitch, and same approach to every subsequent customer you meet. Your calendar would certainly be very busy, but you’re also leaving a trail of confused customers. There is a lot of activity, but there’s zero iterative processes and it wouldn’t be very productive overall.

A friend of mine once worked for a startup where the founders would constantly repeat their approach to traction in what felt like a cruel version of Escher’s Penrose stairs. They went from approach A, saw it fail, then jumped to B, failed that, and then went back to A. In the entire period it constantly felt like they were busy and productive; the calendars were always scheduled regularly with calls and meetings, but their approach didn’t evolve at all after 2 years. In the end, the lack of adaptation and the inability to really iterate and change led to them closing down after they ran out of funding.

The same observation can be seen in development teams as well. I’ve come across sprints where every cycle features a torrent of activity but no forward motion. When I was engineering manager over company X, the developers there had been wrangling with the same bugs all the time. These were serious issues - duplicated transactions, performance issues (read: 100 requests per minute crashes the whole MVC application) among many others. Bugs were reported, and some of the higher priority ones were added to existing/upcoming sprints. The team made PRs and marked the bugs resolved. Except they weren’t. The same issues would creep back, be reported again by existing users after some time. After 2 months, and many supposedly fixes merged, the bug count didn’t decrease.

I’m still learning to iterate better as a software developer, and help others move through that process better when I’m working with a team. Processes don’t always ensure success, but adopting one tends to better your odds. And they provide a framework for me to keep on learning, despite the outcomes.