This is part two in the series of two posts, discussing the Technology perspective. Missed part one? Find it here.
We’ll in this post discuss the Technology perspective in IT Projects.
The technology perspective - a double-edged sward
The technology perspective is not only a deep rabbit hole – it’s a double-edged sward on the one side offering nearly unlimited solutions to (digital) problems.
On the other side we’re presented with a growing cognitive load. Even the brightest and hard-working full-stack developer has no chance to stay on top of all technology permutations. This (potential) cognitive overload can and has to be handled.
IT as business enabler
Here are just a few of the opportunities and challenges faced by today’s IT-organizations:
- Facilitate a strong alignment between autonomous teams
- Facilitate encapsulation of complexity like services (see e.g., Backstage) and operations (like Serverless)
- Making it easy to do things the best way and as standardized as possible (Spotify uses the notion of The Golden Path)
- Automate processes like Continuous Integration / Continuous Deployment [CI/CD]
- Facilitate transparency with e.g., an Objectives and Key Results [OKR] system and various other metrics
- The world of Cloud Computing offers an explosion of advanced services for the IT solutions themselves and for a fine-grained API-economy
- Often we’ll have to manage a multi-context environment with e.g., AWS, Azure, on-prem Cloud (like Open Stack), ERP, PLM, MES, classic “monolith” legacy systems …
- Cloud architectures can be based on (a lot of) microservices requiring e.g., a Kubernetes orchestration
- Container technologies like Docker provide portable application development
- …
The challenge with cognitive (over-)load
Classically an organization would do development in one branch, “throwing the result over the fence” to another branch doing operations.
A more modern approach is known as DevOps (a contraction of Development and Operations) having a team own the whole ting. The principle is “you build it – you run it”.
Using DevOps we avoid “throwing the result over the fence” thereby increasing the speed-to-operation.
But – we’re still using time in the DevOps team to handle operational complexities related to the operations infrastructure.
Note: Because (cyber-) security is a growing concern, the trend is towards DevSecOps.
DevSecOps represents a natural and necessary evolution in the way development organizations approach security. In the past, security was ‘tacked on’ to software at the end of the development cycle (almost as an afterthought) by a separate security team and was tested by a separate quality assurance (QA) team.
The Holy Grail is to have the developers spend as much time as possible on developing value for the business / customers and as less time on infrastructure as possible.
Virtualization (like virtual machines and container technologies) and Cloud technologies have improved that a lot with e.g., modelling infrastructure in configuration files. This is call Infrastructure as Code [IaC] (using e.g., YAML files). The infrastructure code is under version control and integrated in the full configuration.
The DevOps team thus “only” have to configure the operations infrastructure as code, but it still needs a deep understanding of operations dynamics for it to create and maintain the infrastructure code.
Ultimately the DevOps team can spend little time on operations and focus on developing code.
5+ years ago a new computing paradigm saw the light of day, and was coined the Cloud Serverless Architecture. Not that there are no servers, but that the operations complexities became fully encapsulated. All the major Clouds providers (Amazon – AWS, Microsoft – Azure and Google – Google Cloud) have and mature Serverless services.
A simple Cloud architecture using AWS’ Serverless compute service, Lambda, is shown to the right. Understanding the full implications of Serveress is a big topic in itself, but here are some highlights:
- 100% operations encapsulation (though you of course still have to design and optimize your application not to consume more resources that needed!)
- pay only when used
- automatic service provisioning and scaling – no worrying about capacity provisioning and de-commissioning
- ideal for burst-usage and / or for unpredictable usage patterns
- can potentially become very expensive to use if you’re not understanding, how your application consumes resources
Serverless is not a Silver Bullet. Its is one of the options to consider. Depending on the usage-profile, a non-serverless architecture can be a better / cheaper choice.
The Internal Developer Platform
The technology in itself – as e.g., an Internal Developer Platform [IDP] – can help
- gain and maintain an overview
- do things the best, most efficient and standardized way
- encapsulate complexity
Two examples of an IDP are
To be concrete the following will use Backstage as the concrete example of an IDP.
Can you spare the time, this video helps understand an IDP in general and Backstage in particular.
Today [November 2021] Backstage has gained momentum having 50+ public adapters. Rather impressing!
As you might recall from part one, the Platform Team is one of the four suggested team-types. Some companies are taking it one step further seeing the platform as a product (yes – with a Product Manager) clearly signaling its importance.
Have a look in EMPOWERED.
Marty Cagan, EMPOWERED page 222
IDP (Backstage) - why use it?
The infrastructure should be abstracted away from you, enabling you to focus on delivering code / business value.
Backstage is a one-stop-shop to find
- all your tools
- all the technical documentation
- all the teams – who is doing and being responsible for what
Backstage helps
- standardizing the way software is created in an opinionated (good practices in Spotify) way supported by templates (e.g., how to create a micro-service) – a one-click fully functional microservice with everything like CI/CD, documentation etc.
- recuse the same technology stack – making it easy to do the right/best thing – also called a Golden Path
Spotify
IDP (Backstage) - what is it?
Backstage has three main areas
- Service Catalogue
- Service Definitions
- Integrated Tooling via Plugins
As explained in the video above, Spotify’s ambition with Backstage was/is to provide a “single pane of glass” for the developers. One of the neat tricks is the growing portfolio of tooling plugins allowing the developer to stay in Backstage (e.g., not having to context-switch between a multitude of systems / platforms).
Modern architectural patterns
The modern IT architecture
- uses a microservices-based distributed pattern
- uses a combination of sync / async-driven communication (e.g., Apache Kafka)
- will automatically up- and down-scale infrastructure capacity
- is build on redundancy with automatic recovery
- supports a fully automated CI/CD pipeline
- wants to keep things as simple and fast as possible (e.g., preferring REST over SOAP)
- supports an easy-to-use service-discovery
- provides transparency and debugging capabilities into operations, including easy-to-rollback
- provides a rich environment on which to build an API economy and communities
These modern patterns are standing on – and having evolved from – past good practices like
- the Enterprise Architecture Integration [EAI] bus with centralized business logic
- distributed computing using CORBA, SOAP, XML ..
- message-queuing like IBM’s WebSphere MQ
Wrap-up
More that ever, IT development is a team-effort for the simple reason that we as individuals do not stand a chance knowing everything needed to bring a solution from idea to full production.
The good news is that technology in itself helps us orchestra this delicate process.