Slow is Smooth, Smooth is Fast: Agile and Predictability

Long thoughts on project administration and the micro decisions that make good software projects

Let's take a very stylized view of two software projects, one unsuccessful and one successful. A stylized view of an unsuccessful software project would look something like this:

This software project was a mess. We went far over the estimated time to complete it, went far over budget, and everyone was burnt out by the end. Our business leaders now wish we hadn't started, had outsourced the whole thing, or are outright considering writing it off. Because of the stress of this project, our (customers/developers/vendors) are furious, and we've lost or damaged relationships with many of them.

I've heard something to that effect, and lived it, on several projects. Here are some (also stylized) reasons for why a project could have that outcome:

  1. Over time/budget: If you're a business leader, this is ultimately the worst part. If you allocate $50mm to a project intended to impact your revenue by $75mm, a 50% miss on project scope – not uncommon in software – makes it not worth your time.
  2. Stressful: Regardless of success by business metrics, human beings can only sprint for so long, and once the organization is tired it can be hard to recover. A project that makes a team stressed out and frustrated will feel like a failure even if it's a quantitative success. Worse yet, once deadlines start being missed, it's almost impossible for leaders to feel excited about giving a team a break to reset and rest, even if one is sorely needed.
  3. Wasteful: What good is a software project you have to write off, or whose output you abandon? Compounding the short term financial risk is long term reputation, monetary, and culture risk: Developers don't want to work places where their work is thrown away, business leaders don't want to put their names on failed projects, vendors don't want to be blamed for lack of delivery, etc..

Now let's look at a stylized example of a successful software project, something I am thankful to have lived through a few times also:

That project was awesome. We finished about when we expected to, and basically on budget. Our users loved it, and we think it's going to be a big driver of growth/savings for our company. The best part is, while we definitely had some late nights, the team and our partners felt energized and excited throughout. Now we have a lot of great ideas for V2.0 that we can't wait to get started on!

And a few key reasons why that might be the outcome:

  1. On time/budget: What better outcome is there for business leaders measuring success in profit margin and risk level? Coming in at expected cost and on-time means no worrying over how this project fits into your financial model for the year.
  2. Sustainable: While no project is immune from crunch entirely, the amount required was low enough that people working on it experienced good work/life balance.
  3. Valuable: The work that was done was meaningful, and has impact. Everyone can see that their time was well spent, or at a minimum, not wasted.

Sustainability

Okay, those are admittedly pretty generic and simplistic examples. But let's ground those in reality:

Measuring Developer Productivity: Real-World Examples
A deepdive into developer productivity metrics used by Google, LinkedIn, Peloton, Amplitude, Intercom, Notion, Postman, and 10 other tech companies
The Pragmatic EngineerGergely Orosz

In this post over at The Pragmatic Engineer the authors surveyed many tech companies about their metrics for determining developer productivity on real software projects. Some companies used ease of development (DX) metrics primarily, others measured successful deploys, still others used qualitative surveys over quantitative metrics. Yet common to them all were measures of project sustainability – how effective are our changes at the current development pace, how happy is the team at the current pace, and how long can we sustain that pace? That measure of sustainability is the core of what makes a software project more like our stylized "good" one, and less like our stylized "bad" one – a good software project is predictable.

So if good projects are predictable, and predictable projects have high sustainability (and thus productivity) how do we make software projects more predictable?

Agile and predictability

One key thing about Agile/Scrum1 is that while it is not fair to characterize it as a religion, it is correct to say that it is a belief system. Like other belief systems – artistic, religious, political, etc. – it is almost impossible to say broadly that someone has "correct" or "incorrect" beliefs. Two rational people can value different things, and thus never agree on specific beliefs or rituals. I certainly don't think that I know how to do "good" Agile, and others only do "bad" Agile. But I have worked on teams that were unambiguously painful, demoralizing, or difficult to work on because of their working style, and often the core issues stemmed from the team or the leadership valuing things that were counterproductive to good product output. Good product output stems from predictability, and it is my core belief that what makes Agile actually valuable is the ability (though not the guarantee!) for Agile projects to be substantially more predictable than projects run on waterfall/scrum-a-fall methodologies. Why is that the case? For that, we have to take a closer look at the day to day practice of an Agile team.

Day-to-day sustainability

Let's say in the middle of the sprint you have a developer working on a 5 point ticket2, and while that ticket is in progress, the product owner decides that this ticket is no longer a priority. In fact, it's so low priority that the developer working on it should stop and switch to a 3 point story that was either just written, or pulled in from a future sprint. The PO feels bad about the change of direction – nobody likes context switching – but also believes strongly that the team's time is better spent on the new work item rather than the one in progress. Is the PO right to make the change? Or should the change in priority be reflected in the next sprint instead of the current one to preserve the sprint plan?

Generic example of how these tickets might look

Assuming that the original 5 point ticket isn't invalidated2 , another way we could ask that question is: what is the switching cost of moving from the in-progress 5 point ticket to the new 3 point ticket? Here are some theories:

  • The switching cost is zero: a 5 point now is roughly the same as a 5 point next sprint and vice versa. In an average velocity view (foreshadowing) it will all shake out in the end.
  • The switching cost is too high to consider or wrong to pay: Switching out stories is anti-Agile because it breaks the sprint plan, and you should never change the sprint plan mid-sprint because it's "bad Agile."
  • The switching cost is somewhere in between: There's definitely a cost to switching stories between sprints and changing the sprint plan, but there are definitely times that cost is worth paying.

Agile-agnostic folks or detractors will likely prefer the first approach (always be flexible), while Agile dogmatists will likely choose the second approach (never change the plan, perform all Agile rituals with finality.) Unsurprisingly I think the practical answer is the third approach, ("it depends,3") because I believe is possible to roughly quantify the cost. For example, here are some components that might make up the switching cost for user story:

  • Context switching: It literally takes some time for the developer working on something to save their work, close the browser tabs they were referencing, etc.
  • Communication/meeting cost: The decision process to change priorities mid sprint requires conversation offline and online, including potentially meeting time
  • Documentation cost: If you've been working ahead on the sprint demo slides, or technical documentation for this sprint, you may need to adjust either of those places to reflect the new change

My rough ballpark for the combined cost of each of these factors is 30-50% of the story point assigned to a story. So in the case of our 5 point story, the switching cost is probably between ~2-3 points assuming a Fibonacci point scale. If your team's average velocity is 20 points, that may mean a miss of ~10%. If your team's velocity is more like 50 points, that might mean a miss more like ~6%.

Ok, so how do we make that decision? Is that additional cost worth it? In order to answer that, we have to go back to the macro view.

Putting it together: Average Velocity

Generally speaking absolute time estimates in Agile are frowned upon. For my part, I personally have never seen a project succeed when the Agile team is required to give absolute time estimates on work. But business leaders still need to give stakeholders an idea, even if an imprecise one, of when a new feature or program will be done.

Thankfully, even with relative estimates, you can still easily achieve a relatively precise estimation of work by using Average Velocity – assuming you're willing to make the necessary tradeoffs.

Illustration of Average Velocity (green line, here roughly ~47 points)

Average velocity is the average number of points per sprint completed by an Agile team. It's a powerful metric if used correctly, because it represents the actual output of your engineering team over time. Once you have your organization's average velocity, when you have a new program or feature that your architects and engineering leaders have already broken down and estimated4, you can use that measurement to give a narrow time estimate for its completion.

Say for example you have a new initiative that your engineering leadership has estimated at 150 story points. If you have three teams which do 30 (standardized5) points each per two-week sprint, then you can say that initiative is likely to be completed in about 5 sprints, which is 10 weeks. Add some buffer time to handle the unexpected, then account for PTO or holidays, and now business leaders can say "our go-live date for these features is 15 weeks from today" with a high degree of certainty that they're going to hit that date, because that timeline is based on the organization's proven capability.

Tradeoffs

Like anything else though, proper usage of average velocity requires some tradeoffs. Primarily, it requires you to hold several aspects of your organization as constant as possible to improve the likelihood that the average velocity you compute is based on the team likely to execute the future work measured against it.

If you have an Agile team who knows each other well, and consistently delivers 30 points per sprint, that velocity will change if several team members are replaced or rotate out. When that instability occurs, the velocity of the team will necessarily fluctuate as new team members get onboarded and learn the systems and norms, and it may settle at a higher or lower number than it was before. Similarly, if a team's responsibilities change suddenly – say a frontend team now has to work on backend services, or vice versa – it will take them time to accurately estimate in their new problem domain6, and average velocity will be unreliable until stability returns.

But how many software engineering orgs are willing to commit to stable teams, with stable responsibilities, and give them time to settle into a groove? Many software organizations rely on heavily outsourced or contracted workforces, often, ironically, for "flexibility." But that staffing flexibility directly compromises project predictability, as it causes constant fluctuation in average velocity, and program planning becomes more difficult as that key metric becomes unreliable.

There's no way around the fact that team stability is the key factor to ensuring accuracy of estimation, and team stability requires program stability. Organizations have to make long term, considered decisions on software projects, and commit to constant, steady iteration rather than allowing sudden upheaval in order to maximize predictability.

And, answering our micro questions

So back to our 5 point story in progress, and our 3 point story it might be switched out for. What should that team do? Well, focusing on the key value we identified – predictability – here are some options and reasons to choose them:

  • Keep the sprint plan the same: If the current story in progress is still valuable work and hasn't been invalidated, then the PO and engineering leader should think long and hard about the necessity of this new story. Is 5 or so business days going to make a real difference to the wider product? In a predictable and stable engineering organization, it seems unlikely that any one story would be that important.
  • Break the existing story up: Breaking down stories into really small, atomic pieces of work is an art unto itself and deserves its own discussion. If the 5 point story in progress can be broken up, then we can effectively reduce the switching cost (30-50% of the points) because there will be a natural stopping point the engineer can use. This may even cut down on documentation or meeting switching cost, because it might be sufficient to say something was "partially completed this sprint" in your demo deck or documentation. Again though, if you're making each story highly atomic, this shouldn't be possible most of the time.
  • Recalibrate future sprint plans: In the event that switching out the story does make sense, and that becomes more common, then one way to handle the switching cost is to adjust your sprint plan to accommodate the "true" velocity. For example: if your nominal velocity is around 30 points, but you're constantly incurring switching costs, your "true" velocity may only be something like 20 or 25. This will manifest in constant carryover in tickets between sprints, and likely frustration from business stakeholders. Accounting for that buffer by deliberately planning sprints against the "true" velocity will mean less broken promises and higher stability.

Slow is smooth, smooth is fast

In my view, a properly run Agile project focuses on long term sustainability and predictability, and that focus creates measurable accuracy in estimation. It's up to each team to evaluate their predictability decisions carefully, and up to project leadership to preserve team's autonomy to do so. Aside from making projects easier to administrate, a focus on project sustainability makes it more enjoyable to work on; a true win-win for both business stakeholders and frontline engineers.

But organizations must accept the tradeoffs required to achieve that stability, namely adjusting staffing and retention models – not and easy sell. Predicability in software projects requires commitment to long-term thinking and investment into engineering organizations. Many firms will struggle with that truth, especially ones new to digital projects. But ultimately, the benefit of being able to reliably deliver digital projects near the estimated date of completion is well worth the cost. By shifting the risk from estimation/project management risk to product risk, business leaders can refocus their time where it is better spent – product strategy – and engineers can focus on consistent, high quality delivery.

Footnotes:

1: I'm typically quite loose with the terms "agile" and "scrum" – as I think most people are. There's probably a good technical distinction, but I think it's not particularly important to this discussion.

2: Another place I'll be loose with terms: I'm going to use "ticket" and "user story" interchangeably. As mentioned in a previous post, I'm not a fan of the common "user story" format generally, and I think it's easier to just think of a ticket or user story as an "atomic unit of engineering work" for the purposes of this post.

3: I am a consultant after all.

4: A major assumption in my suggested approach is that your architects and engineers are good at relative estimation! One way to ensure that is the case is to make sure that your engineering leadership is made up of people from within the working teams – that is to say, having an architect moving from domain to domain, or product to product, makes it less likely that architect will have anything precise to say about any one estimation. Furthermore, the better the engineering leaders know the exact mechanics of the systems being modified (by, for example, working on it daily) the more precise their estimation will be.

5: When doing program level estimation, I generally think it's better to standardize your velocities – that is, take the average/median/etc. velocity of all teams as your overall program velocity.

6: One way to reduce this cost: organize teams such that each team has experts on the different layers of the stack. If every team has a cloud engineer, a frontend engineer, a backend engineer, and an AI/ML engineer, then in theory they'll be more flexible to any of those domains than say, a purely backend team. This isn't always feasible, but on small to medium projects can be highly beneficial. I always try to organize my teams such that all responsibilities are shared, and people are forced encouraged to rotate problem domains regularly to stay sharp.