Balances and the forces of complexity

The balances and the forces of complexity are all around us:

Spinning top

With the constant changes in technologies and markets, organizations never find a steady-state. What was a working balance yesterday, might be thrown out-of-balance tomorrow.

Or put into other words: the balance needs constant adjustment not to fall over like the spinning-top needing maintained spin. Our job in our organizations is to understand and maintain these balances in ways working for us in our unique context.

The ambition with this post is to discuss the technological and organizational balances and the forces of complexity

Types of complexity

In an InfoWorld article you’ll find Justin Etheredge, cofounder of the software agency Simple Thread, provide the differentiation between “essential complexity” and “accidental complexity”:

Essential is the complexity in the business domain you are working in, the fact that enterprises are extremely complicated environments, so the problems they are trying to solve are inherently complex. 

The other area is accidental; this is the complexity that comes with our tooling and what we layer on top when solving a problem.

Justin Etheredge in InfoWorld, November 1st 2021

There is no reason to believe that the “Essential complexity” will flatten-out or even decrease. Global market forces will keep fueling it. What we can do, to some extend, is influence / mitigate the “Accidental complexity”.

Roughly speaking we have a few options available, dealing with “Accidental complexity” like

    • the way we organize and facilitate work in e.g., networks of highly aligned but loosely coupled teams 
    • encapsulate knowledge e.g., providing a higher-level of abstraction
    • enforce standardization and whitelisting of a narrowed-in technology selection

The notion of complexity encapsulation with higher-levels of abstraction is not new. It is all around us

    • when you turn-on the water tap
    • when you connect your computer to the internet
    • when you turn-on the engine in your car
    • when you flip the light switch
    • when you enjoy the inflight meal
    • when you compile a piece of source code
    • when you send an email
    • when you open a bottle of beer (yes – a lot of high-tech is behind a bottle of beer)

Our society and way of life is based and dependent on these high-level abstractions enabling us to leverage the vast underlying complexity.

Trend: explosion in IT permutations and feature-expectations

As discussed earlier, the IT-developers of today are facing a situation

    • on one side having more tools, services, frameworks etc. at their disposal than ever before. As Camille Fournier (head of platform engineering at Two Sigma) said in an interview with InfoWorld:

You are like a kid in a candy shop in the Cloud.

But as you grow and try make things fit together, the complexity absolutely multiplies.

Camille Fournier, head of platform engineering @ Two Sigma

    • and on the other side extreme challenges navigating the complexity explosion:

The shift from building applications in a monolithic architecture hosted on a server you could go and touch, to breaking them down into multiple microservices, packaged up into containers, orchestrated with Kubernetes, and hosted in a distributed cloud environment, marks a clear jump in the level of complexity of our software.

Add to that expectations of feature-rich, consumer-grade experiences, which are secure and resilient by design, and never has more been asked of developers.

InfoWorld, November 1st 2021

Encapsulation of technology complexity and higher-level abstraction

Just as you cannot expect the normal driver to understand all the complexity under the hood making the car move, you cannot expect the individual developer to know and stay updated on all the technological details in a high-tech product.

We need as individuals and organizations to decide on the balance between knowing all the intricate details and “just” being able to leverage the technology via higher-levels of abstraction. Again – this is not a new concept – even for highly skilled developers. E.g., an IT developer will use

    • C# (normally) without needing to understand exactly how C# is implemented
    • libraries, Cloud services etc. via APIs

But – abstracting each piece of the puzzle is no longer enough. Especially with the explosion of Cloud-services (see e.g.,  AWS’ Marketplace or the vast number of CNCF open-source projects), we need consolidated abstractions across abstractions. This multi-level abstraction is exactly on of the purposes behind an Internal Developer Platform [IDP].

In The IT Project part two – seen from a Technology perspective two examples on IDP were shown. Humanitec, being one of the IDPs, provides a balanced discussion on the IPD in the post What developer self-service shouldn’t look like. As cited below, it is (as always) a matter of striking the right balance; right meaning what is best in your specific context.

Developer self-service is the best thing that can happen to a team. “You build it, you run it” is what you would want for your engineering team. It’s setting the right incentives, unblocking developers, reducing cognitive load. Developer self-service means striking the right balance between overwhelming people by letting them operate everything and restricting them by building abstractions. 

Humanitec BLOG, August 27th 2021

Trend: the never-ending stream of framework / method Silver Bullets

Since Frederic W. Tailor, Max Weber and Henri Fayol we’ve seen new Silver-Bullet attempts again and again to harness the wild beast: how people work together. Add to that the never-ending list of “management literature”. My own humble list is just a tiny fraction.

This is of course great for the consulting industry and let’s face the uncomfortable truth: this will continue forever for the simple reason that you cannot put human behavior and interaction on a formula. 

Which is best (you’re right – a nonsense question)?

    • PRINCE2 ©
    • Scrum
    • SAFe
    • Scrum@Scale
    • Kanban
    • Half Double
    • Update: AGILE FLUENCY® MODEL
    •  …

or – here we go again – the latest Silver Bullet: FAST Agile from Fluid Scaling Technology:

FAST AGILE

Made in and for the Age of Disruption.

FAST harnesses the same creative organizing force of the universe itself
– self-organization.

FAST combines Open Space and Open Allocation to create a lightweight, simple to understand, and simple to master agile method – that scales!

Fluid Scaling Technology

In addition a pleasingly small (19 pages in version 2.1) FAST Guide is provided. How cool is that. For those of you like me being a big fan of Marty Cagan, FAST is seemingly directly facilitating the product-led approach detailed be Marty Cagan:

FAST aims to cater to Product Development as defined by Mary Cagan in his book Inspired. This represents a different way of working for software developers used to other now-traditional agile methods that are heavily focused on delivery only.

“A big part of the concept of product teams is that they are there to solve hard problems for the business. They are given clear objectives, and they own delivering on those objectives. They are empowered to figure out the best way to meet those objectives, and they are accountable for the results.” (ed.: quote from INSPIRED)

Fluid Scaling Technology

And yes – you guessed it: FAST comes with new definitions of values, roles, artefacts, meetings, principles for scaling …

Sorry, not even this latest of the greatest – FAST – will save you with the perfect balance for the same reason that all frameworks / methods fail: you cannot codify human behavior and interaction. 

In this article from Agile Alliance, Ron Quartel actually attempts such a codification:

I tried to codify the system, and we named the system Fluid Scaling Technology with the acronym FAST.

Self-Organization Eats Agile at Scale for Breakfast section 7.2 by Ron Quartel @ Agile Alliance

Encapsulation of organizational complexity and higher-level abstraction

Force frameworkYou can chose the seemingly easy route and adapt an already-made cookbook prescription forcing a framework over your organization. This is what has happened in a significant number of companies going all-in with SAFe.  

Just recently I attended a network meeting around Agile with representation from big (and successful) high-tech companies having attempted just that: adapted SAFe “by the book”. One of the representatives made it crystal clear:

We do not want or have the time to survey all the frameworks/models available and adapt relevant parts – we use the SAFe roles, definitions, processes, artefacts etc. explicitly to form our organization and agile way of working” 

Let’s call this the “Brute-Force” approach.

Alternatively you can chose to do your homework, understanding the growing number of frameworks / models and have that insight combined with a detailed understanding of your unique context. Let’s call this the “Cherry-Picking” approach.

No matter what approach you chose, the purpose remains the same: encapsulate organizational complexity in a way enabling us to operate on a meta-level … the higher-levels of abstraction.

As outlined earlier, strict processes can be very helpful but also harmful:

Good process serves you so you can serve customers. But if you’re not watchful, the process can become the thing. This can happen very easily in large organizations.

Jeff Bezos

First, I can’t help but notice that so many of these process people have never performed either the maker role or the manager role in a strong product company.  They have more enthusiasm than experience or knowledge.  Not only does this put exactly the wrong people in control of how teams work, but these people are prime targets for the armies of consultants and vendors that have figured out that these people are their best path into your company.

Second, in many cases, these process people are hired because the managers are not willing or able to do their job – especially in terms of coaching and developing their makers.  If that’s what’s really going on, then that is the problem that must be fixed.  A process person is a very poor substitute for a strong manager with the necessary expertise.

Third, process is a lot like religion.  People get fanatic about their favorite processes, and it’s very difficult to reason with a fanatic.  (As a side observation, my theory is that since these process people aren’t makers or managers, they tend to have more free time than the rest of us, and many tend to spend a good portion of that time on social media preaching their particular religion).

Marty Cagan, October 2021

Wrap-up

Technological and organizational complexities are here to stay for good and bad. What differentiates you from your competitors, is your ability to find and maintain the balances unique to you in a way best serving your business.

Projekt.dk