Vertical Slicing

Delivering working software in the hands of the customer is one of the core principles behind agile software development, and one that we follow at Webjet. This means teams break down their work into bite sized chunks and deliver working chunks iteratively.

But back when we started our agile journey, this usually meant working on the back end, followed by working on the front end. As you might have guessed it the “working software” was only delivered at or close to the very end. Not ideal.

I didn’t know the word for it at the time, but we were doing Horizontal Slicing, i.e. breaking down work based on its functional area, without taking a holistic end to end approach. We were iteratively delivering a car by delivering only the wheels first.

MVP

Going back to the core principles, we looked at how we can deliver a true MVP, something that provides value to the customer and to the company, something that we are happy to have in production. We came across the concept of vertical slicing, which helped us articulate the MVP approach.

Vertical and Horizontal Slicing

Using the example of creating a new flight search experience for mobile browsers, I will share our thinking and how we approached vertical slicing, which will help crystalize the concept

Our flight search experience for mobile needed an update. The incumbent design was a copy of the legacy app experience, built using JQuery mobile integrated into our monolith. This made making changes more complicated and take longer to deliver. It was not aligned with our strategy of splitting it into independently deployable microservices.

Old Mobile Search

We decided to create a new React based service to render the responsive front end coupled with a new search API. This would make iterative changes less complex and faster to market. But there were a lot of features to deliver, multiple result views, and the integration with the legacy platform.

New Mobile Search

To help define the workflow the team documented the high level workflow

High level workflow for mobile Flight Search

To help guide our decision in defining the MVP we looked at the following

  • Data on the booking mix i.e. domestic vs Trans-Tasman vs international, one-way vs return, passenger mix etc
  • Key path dependencies
  • An MVP that delivers value to the customer

The following were the results of the analysis

  • A large majority of the bookings were domestic, so the real value from a customer’s POV lay there
  • As domestic volume was higher any issues in the new path could be amplified
  • A wide range of passenger mixes existed, e.g. 1 adult, 2 adult and 1 child, 2 adults etc
  • Integrating into the legacy shopping cart was a key dependency. Until now the search and booking happened in the same platform.
  • The incumbent mobile design was quite dated and had limited functionality. Customers could only sort on 4 attributes and there was no filtering

Based on this we determined the first slice as a

  1. Single one-way adult search: limits the scope of the customer cohort
  2. For a medium capacity route: reduces the risk of unexpected failures and provides value to the customer
  3. With just the sort functionality currently supported: limits the feature set that needs to be built
  4. That can be searched and booked end to end: reduces key path dependency

MVP identified in Mobile Flight Search

The team went on to develop this MVP and release it to the customers in a fraction of the time of the full design build. After based on feedback from data gathered, we iteratively added more features starting with supporting return search and result filtering. The team could quickly support more routes and passenger mixes using feature flags as their confidence grew in the new services.

Not only were we able to deliver value iteratively, but since we only focused on a narrow path it gave other advantages

  • Validation of the new design in the hands of the customers
  • The team did not need to develop a Trans-Tasman or an international design till much later
  • Increased confidence in monitoring and alerting of the service as the load slowly increase

Looking back at our approach the key element that made this a success was a strong engagement with the business stake holders. Having their input and buy in to delivering a fraction of the promised functionality in the customer’s hands was crucial. We used real metrics to measure customer satisfaction with our new design. It also helped us tweak features that were causing issues and iterate on the features that were well received.

Without it we would be relegated to releasing in our UAT environment which would not give true confidence in the design, and customer would have waited months to get any value.

If you are interested in joining our team, we are hiring!
Our current job postings can be found here: https://lnkd.in/gycBPXPx

Monolith to Micro-services | Part 2 | All aboard with Kubernetes

As the paradigm shifts more to container workloads and microservices, Webjet was looking for a way to deploy containers as well as manage them. In part one we dived into the journey of microservices, our traditional Azure Web App architecture and how we started adopting container workloads. We learnt to write systems in golang and dotnet core, how to write Dockerfiles and build up a series of base images. Most importantly we built the foundation of what’s required to build and manage containers effectively.

This era of our container journey plays a big role in how things turned out. When we started looking at container orchestrators, there were only a few and not all of them were production ready. If you read our blogs you should know by now that Microsoft Azure is our “go to” platform, so it is where we started. At the time (late 2016), the most production ready platform was DC/OS . Kubernetes was not released yet and Docker Swarm was in private preview. For us, the orchestrator needed one key feature..

Run my container and keep it running!

The main challenge was building a CI/CD pipeline that would plug into a container orchestrator and have a service discovery mechanism, so we could route traffic from the customer’s browser, to the individual containers, no matter where they were running. We wanted to be platform agnostic, so we could run on any orchestrator. The good things about every orchestrator, is that they generally provide built in service discovery and have a method of defining an “Ingress” (Where network traffic enters) through a public IP address.

Batman’s Operating System

For DC/OS, it was Marathon and NGINX:

It serves the purpose of “Ingress” and has a public IP address. Customer traffic arrives at Marathon, and it can find other containers inside the cluster without having to know private IP addresses. Marathon routes traffic to our own customised Nginx container, which in turn serves as the API gateway. The API gateway routes to the correct container based on its URL path and terminates SSL traffic and sends traffic privately to microservice containers.

Ask Jeeves

To solve the CI/CD piece, we turned to the popular Jenkins build tool. One key feature that Jenkins provide is ability to write pipeline as code .

Writing a declarative pipeline definition as code allowed the team to have version control for their CI/CD pipeline side by side with the code. It also means no one must manually create pipelines with the web user interface. Pipelines can be parameterised and re-used across new microservice implementations. This allows us to move faster when implementing new services and we don’t have to spend time designing the pipeline from scratch.  The Pipeline file defines the CI/CD process and the Dockerfile defines the application and its dependencies. These two files form the ultimate source of truth and allows for a fully automated deployment platform where the source of truth is in the source control repository and not in the snowflake environment.

Once we had these two components in place, CI taking care of the image building and pushing to Azure Container Registry, CD taking care of deployment to DC/OS and Marathon taking care of service discovery, we had a foundation in place to deploy our first production service.

Webjet chose a small, isolated, non-critical piece of functionality which we pulled out of the legacy monolithic stack and containerised. It became the canary that would test out the container CI/CD and orchestration system.

One thing we were not satisfied with, was the lack of secret management in the open source version of DC/OS. This version did not support secret management which at the time was an enterprise-only feature. We wanted to avoid enterprise agreements and vendor lock ins our docker environment. We preferred the ability to lift and shift to various orchestrators when the need arises. Our apps needed to be cloud native, and therefore run anywhere.

Capt’n Kube to the Rescue

Roughly a week into production, Microsoft announced Kubernetes general availability on the Azure Container Service platform (ACS)*. During this time, containers were a new thing on Azure. For us being new to this as well, we were fortunate enough to mature alongside the platform as Kubernetes, which itself was just over 2 years old. We were able to leverage our relationship with Microsoft and worked together with open source teams at Microsoft and share experiences of the journey. Though these close ties we ensured that our roadmap aligned with that of Microsoft and the Kubernetes upstream.

Microsoft alignment with the upstream Kubernetes community and their massive contribution to open source is what got us excited about Kuberenetes. We could finally build a microservice stack on a cloud agnostic and cloud native platform. It can literally run anywhere.

Our next move was to deploy a mirror of what we had on DC/OS, but this time use Kubernetes as the platform. The benefits of our initial CI/CD process were realised, and we seamlessly plugged into the new platform. We replaced Marathon and the Nginx API gateway with Kubernetes Ingress controller. Ingress takes care of service discovery and URL path routing within the cluster. It also runs through a public IP address and operates at the edge of the cluster for receiving inbound customer traffic.

With CI/CD in place we could deploy our non-critical microservice to this cluster and the services were accessible by the customer over the same URL.

The traffic flow looked like this:

[domain.webjet.com.au] —(DNS routing)–> [Azure Load Balancer] —> [Kubernetes Ingress Controller] —(URL routing /api/hotels/upsell)—–> [microservice]

Once tested, all we changed was the endpoint where the domain name was pointing (from the DC/OS IP to the Kubernetes Azure Load balancer IP) and traffic started to flow to the new implementation. We switched over from DC/OS to Kubernetes within a week after we went live. How dope is that?

You’re probably thinking, “how are you monitoring these containers and VMs?”

In Part 3, we will look at logging and monitoring and how Webjet embraced open source tools to simplify the entire monitoring and observability process.

Until next time!

* One thing to note is that ACS was not the managed Kubernetes version (AKS) we know of today

Monolith to Micro-services | Part 1 | An unexpected Docker journey

Micro-services have been a hot topic for us folk at Webjet. Like many other software teams, at Webjet, we have been working over the years with what we would more recently call a monolithic architecture.  Being faced with changing the way we engineer our solutions to meet the scale, agility and momentum our business demands, we turned to micro-services as a solution for delivering new features.   This decision led to an unexpected journey with Docker which we would like to share with the broader community

Where did our journey start, what is a monolith ?

In simple terms, a monolithic application (monolith) is an application where the user interface tier and its data access code are stitched together in a single application on a single platform. As the application grows, more service tiers and application features are introduced and therefore the complexity and configuration of the application and its environment increases. It becomes harder to make even the smallest changes and introducing a breaking change can happen at the blink of an eye. Scaling therefore also becomes a resource-expensive effort, as only whole instances of the entire application can be scaled even though each layer has different load and resources requirements. This leads to over-scaling some layers, but under-scaling others.
So how did we break the shackles of the monolith?
Over time, we started slicing up the monolith into separate services and user interfaces that would become isolated. We would deploy each of these services separately using the Microsoft Azure WebApp platform.

Although this was the first step to a micro-service architecture, we introduced a lot of complexities in our build and continuous integration pipelines. As the number of services grew deployments and setting up CI/CD pipelines took a lot of unnecessary time. We were at a point where we could see that the way we were building and deploying our micro-services would soon hit a bottleneck and slow our journey down.

Where the unexpected journey happened.

Being focused on continually improving our engineering processes, we started a review of how we were deploying our WebApps in our CI/CD pipeline.
WebApps allow you to deploy services on separate servers or to the same pool of servers. Deploying a tiny micro-service to its own server does not sound very efficient. Deploying a few of them sounded like a better option, but what if you have one resource hungry noisy neighbour?

This is where Docker comes in.

Docker is a technology that allows us to build all application code, its operating system, configuration and dependencies into one single entity called a container. For us, it made sense to start building new services on container technology and start the decoupling process of the monolithic application. As a group, we’d have to identify all the components of the monolith that we can break apart, -.e.g flight search, hotel search, shopping carts, up-sell services, autocomplete services, small UI components, etc. , etc. 

For the shift to Docker to happen, we needed a massive technology and mindset shift:, introducing new programming languages, new operating system platforms, new scripting frameworks and many more. 

Docker introduced true immutable applications, the ability to run cross platform and on any cloud. Our Docker solution is also highly scalable and deploys really fast! I can go on and on about the beauties of Docker, but I’ll leave this here if you want to read more on what it is. 

With Docker, we can build layers of images. So as an example, let’s say we use Linux Alpine as a base operating system. Our DevOps team can then build a Webjet certified Linux Alpine image with all the required security patches. This can be used to build a other images that have the dependencies for applications to run, for example, a popular programming language built by Google called GoLang 

We embarked on a mission to start building out our base Docker image library to make it really simple for teams to build micro-services on different languages. If I was a developer and I needed to use GoLang as a language, I can simply build a new Docker image for my application and inherit the Webjet GoLang image. An example hierarchy can look like this:

Now, Webjet development teams can build services top down not worrying about the low level configuration, whilst DevOps and security teams can architect the base images from bottom up to ensure they all stem from the certified base images. This keeps vulnerabilities to a minimum and keep the images lean and small as possible. 

We’ve also utilised Azure Container Registry for hosting our Docker images which makes it easy to start integrating continuous delivery pipelines and deploying micro-services. 

This brings us to Part 2 of this series where we’ll be covering “Container Orchestration” and Kubernetes: How we at Webjet deploy and manage a large number of containers within the Microsoft Azure cloud infrastructure. 

Until next time!