Grading a project on SOLID

I’m a strong believer in the idea that there isn’t such a thing as “domain specific knowledge.” There may seem to be, but at a sufficient level of abstraction my belief is that the domain specific part of that knowledge will go away. And left behind is often, if not always, a nugget of general truth you can take with you. I think about this concept a lot, and I often ask myself “what am I learning right now in this context that might apply elsewhere?” Naturally, given my work as a software engineer, this often means I often apply life lessons to code or find myself taking life lessons from code — the latter more in the code of my mentors than my own.

That question is why, when I was reading this article by Dmitry Non about microservices, I was struck by his description of the ways that principles of code organization can also be applied to team and project organization. SOLID principles are a classic computer science interview topic, and the subject of many an undergrad CS lecture. But at first glance, SOLID principles don’t even apply to all of programming, let alone outside of it. But as Dmitry reviews in his article, they can be a useful framework for thinking about much more than just code:

As we often say, programming is all about splitting a problem into a smaller problems and solving them one by one. However, it’s not only about problems. The same goes for systems and organizations (see Conway’s law). System at every level (including organizational) needs to be sanely split into smaller units. And at every level a well split system units will have the characteristics defined by SOLID principles.

So with that in mind, I thought it might be instructive to apply SOLID principles as a framework to my previous project as a sort of grading criteria. Were there clues in SOLID for how we could have done a better job? What would I do differently, or have done differently. if I started on a project trying to follow these principles as a guideline from the start?

What are the SOLID Principles?

Before we do that though, a quick primer on SOLID principles: SOLID is an acronym that combines the first letters of five principles, each a general rule to follow when designing software or technological systems. Each rule on its own is valuable, but they are typically all remembered and applied together. Truthfully, a technical recap of each would be quite boring, so I’ll encourage you to read the Wikipedia page for each rule in SOLID and instead give them my own descriptions:

• The Single Responsibility Principle: Each component of a system should have only one responsibility. Said another way: “Never half-ass two things. Whole-ass one thing.”
  • Example: Your inventory management API shouldn’t also process payments, and vice versa.
  • Practical example: It’s easier to avoid distraction in a dedicated home office than trying to work from your living room/kitchen.
• The Open-Closed Principle: Software units, broadly speaking, should be open to extension but closed for modification. I like Dmitry’s framing here: “Composition over modification.”
  • Example: Teams should feel free to combine atomic systems (inventory, payments, authorization) but should be very cautious about modifying atomic systems.
• The Liskov Substitution Principle: Any time two systems, teams, or organizations agree to an abstraction, any instance of that abstraction should be usable any where that abstraction is required.
  • Practical Example: You agree to buy 100lbs of tomatoes a week for your restaurant. Any given tomato should be interchangeable. If 20 out of every 100 tomatoes were cherry tomatoes all of a sudden, you’d have trouble making the right amount of sauce. (I guess in this example you own an Italian restaurant?)
• The Interface Segregation Principle: If a software unit has a dependency (another system or library) it shouldn’t break because that dependency has an unstated dependency. 
  • Practical example: If you go to buy a car from me, and my car has a loan on it, you expect me to resolve the dependency of getting the title from the bank and transferring it to you. It would be very weird, and impractical, for me to ask you to sort the whole thing out with the bank instead.
• The Dependency Inversion Principle: Technically speaking it means that code should depend on abstractions, not specific implementations. (Similar to our forklift example.) Again, I like Dmitry’s framing here: “Interface first, implementation second.” Wikipedia helpfully adds that another term for this is “contract-first design.” 
  • Example: The shipping container is a set of requirements of physical size and attachment points to define “this is a unit of stuff.” The shipping container can be built of anything, and may not even be a box. Anything that meets the contract of “shipping container” should be able to be loaded on a container ship. If suddenly a container ship broke because the it was loaded with a single container made of aluminum rather than steel, that would be a poorly made ship!

You can see my thesis leaking into those examples — in my opinion it would be doing a disservice to a potentially non-coding audience to talk about programming design patterns without trying to explain how they could apply to anyone. So, with the SOLID framework as our guide, let’s talk about a specific project and reflect on where SOLID may lead us to lessons learned.

The Project

Speaking as generally as possible, the project was a pretty standard data platform build combined with a pretty standard application build. Data platform was to combine and progressively refine client data from various source systems, and then the application layer would display it to the client’s customers. The initial goal of the project was to provide a client customer visibility into their data at a different level of granularity than existing systems, and to have a modern and easy-to-use interface. My role on the project was to lead a team of 4-10 developers, (BCG, client, and contractor) on the full-stack application build, including the APIs, cloud auth, and user interface.  The primary sources of complexity in the project were the following:

• Client unfamiliarity with the tech stacks involved (Kubernetes, Azure, VueJS, and a BCG proprietary data framework)
• The change in data granularity from source systems (we wanted to think in terms of ‘atoms’, the existing systems were, let’s say, ‘molecule’ focused)
• The self-imposed (as in, literally I imposed it) requirement on the application to deliver a component library alongside the application itself. 

Originally targeted for a 7 month whiteboard-to-release timeline, the application was in a pre-alpha state at 7 months, and closed alpha state at 9 months — pretty good for going from “wouldn’t it be cool if” to “ready for pilot release,” although I’m biased. At the 9 month mark we transitioned to another vendor who took over our MVP to operate it and add a new slate of feature to prepare the application for client-customer release. 

Grading Using SOLID

The Single Responsibility Principle 

On a technical level, we did a great job with this one. Each component of the architecture didn’t really try too hard to do another piece’s job. The cloud architecture piece really was just infrastructure, and there were very few bugs or configuration issues that impacted the ability for the data and application teams to take the cloud work for granted. Unlike other projects I’ve been on involving Kubernetes, I spent maybe a total of 30 minutes out of 9 months tweaking a docker file? The rest of the build and deployment was handled by the cloud folks, and the infrastructure “just worked.” 

On the data side, I interpret this to mean “did the data pipeline end up doing extraneous work, or was it purely focused on data refinement and storage?” Grading in that lens, we also did a good job. The pipelines were focused only on refining data, and while initially there were ideas about expanding the scope of the data workstream to transforming other client processes, they ultimately weren’t executed upon. 

Now for my piece, the full-stack build. Here we didn’t do a perfect job on the whole, although as I’ll discuss later, I think that had more to do with the team structure than the technical requirements. The front-end was remarkably modular — all credit to my excellent team — and we not only delivered the application, but a fully-functional design system and component library as well. Components were isolated, and there was virtually no business logic in the frontend at all.¹ 

Flipping to the people and team side, we took the “single responsibility principle” a little too far in the structure of our teams. The data team was practically, although not formally, firewalled from the application team. And while we tentatively agreed on a code interface between us, that interface wasn’t formalized or developed fully until nearly the end of the project. As a result, it was difficult for the API the application team built to really only have one responsibility, (retrieve data from the database.) At many points in the project we had to bake-in business logic on the API that should have lived on the data pipelines while we navigated communication issues to get those features added by the other team. By the end of the project, the APIs were almost clean of business logic, but it could have been that way from the beginning as I’ll touch on in a later principle.

Lastly, the meta-level: Was the project given a single responsibility upon which to execute? Initially yes, and it allowed for great focus in the first 6 months of the project. Client and BCG leadership was bought in on the specific goals of our project, which were distinct from other client applications and work streams. By the time we were ready for closed-alpha though, we started to deviate from this principle. The initial difference in data granularity agreed upon by the client and BCG was seen as a liability to internal stakeholders. Going back to our atoms versus molecules: there was a concern that many client internal operators, whose references point was “molecules,” would be confused when we only showed “atoms” in the UI. How would they know how to use the application, or would they even support the application’s existence without a “molecule” view? 

Given the change in responsibility from a single data and application focus to two, the end months of my time on the project were much more stressful as we had to try to accommodate both the old and the new feature set, with a data and application codebase designed around the initial single responsibility. 

The Open-Closed Principle or “Composition Over Modification” 

On a technical level, this is one where we had an easy win: The client had three separate systems, each with the concept of a customer “atoms.” Trying to modify or build our features within any one of these would have been the most uphill of uphill battles, so thankfully our project took the approach of composing data from each source system into a unified data model. This allowed us to compose something that paid respect to the unique benefits and drawbacks of each system, while presenting a standardized abstraction at the highest levels of the data model. 

We also did a good job with the composition of client capabilities — they had a few DBAs who were experts in source systems, a few designers, a few product folks, a few cloud engineers, etc.. We augmented each capability, extended with our own talent pool (which, shameless self-plug, is comprised of some of the best in the world) and brought along the client and their teams for the ride. As with any consulting engagement, navigating client culture and politics is hard. But we managed to not only upskill the client counterparts who were bought-in to our ways of working, we also managed to build a new project culture that seems to have momentum even after we left. Had we instead tried to make an existing team work in new ways, or try to get an existing project group to pivot to this new product, it would have been a very tough effort culturally. By composing a new team from across client orgs, and adding our own flavor, we were able to do much better.

The Liskov Substitution Principle 

As stated above, this is an area where we had a lot of room for improvement. Early on, we started talking about “an atom” in the client’s context. But it became clear much later than I expected that not everyone agreed on what was important to say about “an atom.” The data contract for the most important unit of the data model wasn’t built until the application team got down in the data pipelines, almost at the end of the project, to define the contract after months of waiting. 

In retrospect, this was an obvious problem. We knew about 50% of the fields we would need to show in the UI and how they would be computed, but the application team assumed (I know, we shouldn’t have) that one day the data team would present a data contract fully formed to define the rest. Turns out, the data team was thinking the same in reverse! We should have agreed on common interfaces and abstractions early on to ensure everyone was developing against the same waypoints.

Another subtlety: Not everyone in every role was interchangeable. Not every product owner had the same responsibilities, not every lead engineer had the same level of authority, etc.. While this was somewhat natural given the new teaming model and team composition (see previous section), it meant the team was constructed around a few “heroes” rather than baking those functions into sustainable roles and responsibilities.²

On the full-stack application team, I am proud to say I tried and succeeded in requiring, as we said, “everyone to do everything (at least once.)” Everyone worked on the frontend, backend, CI/CD, documentation, Jira ticket writing, etc.. If someone had done 3 backend tickets, I would prioritize them for a front-end ticket and vice-versa. Someone tells me (tech lead + product owner) about a bug or feature idea? Write a ticket for me in our template and send it to me for review³. This cross-functionality of team members meant critical bugs were fixed faster, PR reviews were higher quality, and team morale was higher because our best programmers never got bored. Other teams were not as deliberate in this, and it showed. They assigned work in verticals per person, which caused issues when that person rolled off or, in one case, left the company for a new opportunity. 

In my opinion, “everyone does everything” or “every engineer should be professionally interchangeable” is the key principle that makes a tight team of engineers truly effective. But, your engineers have to be savvy and curious enough for that to work. Not everyone does well with that kind of expectation, or freedom. 

The Interface Segregation Principle

On a technical level, we mostly did well here. This is one where it’s pretty easy to map out if the code has bad dependency issues, and we mostly avoided that. On the data, cloud, and application teams we mostly made sure that dependencies in one realm (Kubernetes, Databricks, Quasar, etc.) didn’t cause headaches for another team. In fact, we made it 9 months before I had to think about how my application was interacting with Kubernetes, because the cloud team did a good job of making it “just work,” preventing my work from “depending” on our Kube configuration.

In a teaming/project sense, there was room for improvement here. Each team maintained a separate Jira board, and there were often requirements to keep client people outside the project team “in the loop” which introduced a lot of people dependencies and red tape. Admittedly, BCG is a pretty loose, cowboy environment on our internal teams — see the “everyone does everything” model above — so our personal tolerance for bureaucracy isn’t super high. But often even client teams were slowed down by a procedural block where they were required to get the blessing of a person outside the project team for something project-related, especially on the cloud side. Lots of churn could have been avoided if the cloud permissions and political structure of our project was isolated from external dependencies from the beginning.⁴ 

The Dependency Inversion Principle or “Interface Over Implementation”

I’ve already touched on this one a bit — we could have done a better job of starting by designing or aligning on contracts for our systems before we started building in earnest. Frank discussions about “what does this entity represent, what fields should it have, etc.” with all specialties on the team: design, development, product, and business. The primary lesson I learned on this project was that “begin with the end in mind” really can’t be applied too specifically. Even just a straw man interface for each of our entities, agreed upon by the tech leads from each team, would have saved so many headaches later.

A place where I was really proud of our work here was in our Jira tickets³ (exciting, I know.) Early on the client asked that we use the “given/when/then” format, but I found that their stories were more written to meet the format than they were to describe work. This struck me as odd — our product folks were quite eloquent and articulate when they weren’t writing in that format. The template we worked off of on my team for the project afterwards was a “context/objective/acceptance criteria” format that encouraged long-form writing. After we switched, suddenly story quality went up; the “interface” of the template was a way to drive higher quality.

I take this principle to mean “agree upon roles early on” when applied to the organization, and in that lens I think we did just ok. While I already touched on the micro level where that didn’t always work out, in the macro view each team proved to be assigned roughly the right set of responsibilities such that the “surface” or “interface” of a team’s responsibilities nested nicely with others. The cloud team did a great job dipping down into app code on rare occasions, and they did so at their “connection point” in the code “interface” (the dockerfile and CI/CD yaml files.) Similarly the app team by the end was dipping down into the data team’s world, creating good contracts and respecting the interfaces deeper in the data model in collaboration with the data team.

SOLIIDifying better ways of working?

How did we do at following SOLID? Unsurprisingly for a technical bunch of folks, pretty ok in the code, especially at the higher levels of abstraction. But when it came to the teaming and organizational level there was room for improvement. Some of that came from the BCG culture, some of that from client culture. 

Again, with the SOLID principles as a framework, what could we do better for next time?

1. Agree on interfaces earlier: There’s no reason we couldn’t have aligned on what the hell we were trying to define, at least partially, early on. From a technical perspective this is my key learning, although the designers and product folks would have had a much easier time of things too.
2. Related: Don’t take the “single responsibility” principle too far, to the point of siloing your teams. I’m going to do more thinking on how technically to do this next time, as my belief is that the only place this can happen is inside the codebase itself, meaning teams have to cross over into each other’s domains or microservices.
3. Align on roles early, and hold the roles accountable: We had a lot of late night and weekend heroics from the BCG team — if we had set out clear roles early on, and defined “the lead engineer on this piece of the architecture must do X, the product owner for Y team must do Z”, etc., we could have saved a lot of late nights and spotted weak points in the process and people earlier.⁵ 

Footnotes:

1. We did end up assigning sentiment to things on the frontend before display. Something that was bad was mapped to “red,” good to “green” etc.. We discussed data-side sentiment, maybe we’ll have time next time.

2. In a consulting context, it’s hard to avoid this. Teams are, by their nature, temporary. Consultants move to new clients and projects. Client team members rotate. That said, if we had put more thought into team design, we might have been able to serve as examples of the clear roles and boundaries we wanted the client to follow as a best practice after we left. I don’t have a great answer yet for the right balance here.

3. Unsurprisingly, this was the hardest one. Getting some of our engineers to flip perspectives and guide a future team member, in writing, through a problem and a potential solution was very difficult for some. I’ll have to write something in the future about our Jira template, I’m quite proud of the thought process it provoked.

4. This isn’t always viable, especially when your team is “composed” from across client silos, as ours was. Not trying to sound naive with “we need less red tape!” but the truth is that it isn’t just your consultants who are annoyed by it. Client folks moaned just as loud when they hit an arbitrary roadblock, or a point where someone halted progress for deniability reasons rather than promoting a culture of trust. Obviously not talking about blockers which are regulatory or ethical, but rather procedural or cultural.

5. The hard part of this of course is not defining the roles, but holding them accountable. In a consulting context (or at least, a BCG context), this is really hard. Client or contractor teams may know each other well, but most likely won’t know the consultants at all, making initial trust building your goal at first. This means that during the critical pre-project time, the working team won’t know each other well enough to have the trust necessary for this process.