The IT Project part two – seen from a Technology perspective

This is part two in the series of two posts, discussing the Technology perspective. Missed part one? Find it here.

We’ll in this post discuss the Technology perspective in IT Projects.

The technology perspective - a double-edged sward

The technology perspective is not only a deep rabbit hole – it’s a double-edged sward on the one side offering nearly unlimited solutions to (digital) problems.

On the other side we’re presented with a growing cognitive load. Even the brightest and hard-working full-stack developer has no chance to stay on top of all technology permutations. This (potential) cognitive overload can and has to be handled.

IT as business enabler

Here are just a few of the opportunities and challenges faced by today’s IT-organizations:

    • Facilitate a strong alignment between autonomous teams
    • Facilitate encapsulation of complexity like services (see e.g., Backstage) and operations (like Serverless)
    • Making it easy to do things the best way and as standardized as possible (Spotify uses the notion of The Golden Path)
    • Automate processes like Continuous Integration / Continuous Deployment [CI/CD
    • Facilitate transparency with e.g., an Objectives and Key Results [OKR] system and various other metrics
    • The world of Cloud Computing offers an explosion of advanced services for the IT solutions themselves and for a fine-grained API-economyPublic API
    • Often we’ll have to manage a multi-context environment with e.g., AWS, Azure, on-prem Cloud (like Open Stack), ERP, PLM, MES, classic “monolith” legacy systems …
    • Cloud architectures can be based on (a lot of) microservices requiring e.g., a Kubernetes orchestration 
    • Container technologies like Docker provide portable application development

The challenge with cognitive (over-)load

Over the fenceClassically an organization would do development in one branch, “throwing the result over the fence” to another branch doing operations.

DevOpsA more modern approach is known as DevOps (a contraction of Development and Operations) having a team own the whole ting. The principle is “you build it – you run it”.

Using DevOps we avoid “throwing the result over the fence” thereby increasing the speed-to-operation.

But – we’re still using time in the DevOps team to handle operational complexities related to the operations infrastructure. 

Note: Because (cyber-) security is a growing concern, the trend is towards DevSecOps.

DevSecOps automatically bakes in security at every phase of the software development lifecycle, enabling development of secure software at the speed of Agile and DevOps.

DevSecOps represents a natural and necessary evolution in the way development organizations approach security. In the past, security was ‘tacked on’ to software at the end of the development cycle (almost as an afterthought) by a separate security team and was tested by a separate quality assurance (QA) team.

IBM

The Holy Grail is to have the developers spend as much time as possible on developing value for the business / customers and as less time on infrastructure as possible. 

YAMLVirtualization (like virtual machines and container technologies) and Cloud technologies have improved that a lot with e.g., modelling infrastructure in configuration files. This is call Infrastructure as Code [IaC] (using e.g., YAML files). The infrastructure code is under version control and integrated in the full configuration. 

The DevOps team thus “only” have to configure the operations infrastructure as code, but it still needs a deep understanding of operations dynamics for it to create and maintain the infrastructure code.

Ultimately the DevOps team can spend little time on operations and focus on developing code. 

AWS Serverless5+ years ago a new computing paradigm saw the light of day, and was coined the Cloud Serverless Architecture. Not that there are no servers, but that the operations complexities became fully encapsulated. All the major Clouds providers (Amazon – AWS, Microsoft – Azure and Google – Google Cloud) have and mature Serverless services.

A simple Cloud architecture using AWS’ Serverless compute service, Lambda, is shown to the right. Understanding the full implications of Serveress is a big topic in itself, but here are some highlights:

    • 100% operations encapsulation (though you of course still have to design and optimize your application not to consume more resources that needed!)
    • pay only when used
    • automatic service provisioning and scaling – no worrying about capacity provisioning and de-commissioning
    • ideal for burst-usage and / or for unpredictable usage patterns
    • can potentially become very expensive to use if you’re not understanding, how your application consumes resources

Serverless is not a Silver Bullet. Its is one of the options to consider. Depending on the usage-profile, a non-serverless architecture can be a better / cheaper choice.

The Internal Developer Platform

The technology in itself – as e.g., an Internal Developer Platform [IDP] – can help

    • gain and maintain an overview
    • do things the best, most efficient and standardized way
    • encapsulate complexity

IDPTwo examples of an IDP are

    • Backstage
      Backstage originates from Spotify having made it available as open-source. It was accepted in the Cloud Native Computing Foundation [CNCF] Sandbox September 2020.
    • Humanitec
      The Humanitec IDP is a fully matured commercial product (with a free Community edition)

To be concrete the following will use Backstage as the concrete example of an IDP.

Can you spare the time, this video helps understand an IDP in general and Backstage in particular.

Backstage public adaptersToday [November 2021] Backstage has gained momentum having 50+ public adapters. Rather impressing!

As you might recall from part one, the Platform Team is one of the four suggested team-types. Some companies are taking it one step further seeing the platform as a product (yes – with a Product Manager) clearly signaling its importance. 

Have a look in EMPOWERED.

An emerging trend is that a growing number of companies are working to manage their internal platforms more like external platform products

Marty Cagan, EMPOWERED page 222

IDP (Backstage) - why use it?

An engineer should not have to be an infrastructure expert to be productive!

The infrastructure should be abstracted away from you, enabling you to focus on delivering code / business value.

Backstage is a one-stop-shop to find

    • all your tools
    • all the technical documentation
    • all the teams – who is doing and being responsible for what

Backstage helps

    • standardizing the way software is created in an opinionated (good practices in Spotify) way supported by templates (e.g., how to create a micro-service) – a one-click fully functional microservice with everything like CI/CD, documentation etc.
    • recuse the same technology stack – making it easy to do the right/best thing – also called a Golden Path

Spotify

IDP (Backstage) - what is it?

Backstage has three main areas

    • Service Catalogue
    • Service Definitions
    • Integrated Tooling via Plugins

Backstage

As explained in the video above, Spotify’s ambition with Backstage was/is to provide a “single pane of glass” for the developers. One of the neat tricks is the growing portfolio of tooling plugins allowing the developer to stay in Backstage (e.g., not having to context-switch between a multitude of systems / platforms).

Modern architectural patterns

The modern IT architecture

    • uses a microservices-based distributed pattern
    • uses a combination of sync / async-driven communication (e.g., Apache Kafka)
    • will automatically up- and down-scale infrastructure capacity
    • is build on redundancy with automatic recovery
    • supports a fully automated CI/CD pipeline 
    • wants to keep things as simple and fast as possible (e.g., preferring REST over SOAP
    • supports an easy-to-use service-discovery
    • provides transparency and debugging capabilities into operations, including easy-to-rollback
    • provides a rich environment on which to build an API economy and communities

These modern patterns are standing on – and having evolved from – past good practices like

    • the Enterprise Architecture Integration [EAI] bus with centralized business logic
    • distributed computing using CORBA, SOAP, XML ..
    • message-queuing like IBM’s WebSphere MQ

Wrap-up

More that ever, IT development is a team-effort for the simple reason that we as individuals do not stand a chance knowing everything needed to bring a solution from idea to full production.

The good news is that technology in itself helps us orchestra this delicate process. 

Projekt.dk