So the Code for this and all future setups can be found here:
https://github.com/infrastructure-as-posts

Today I want to showcase virtual Kubernetes clusters to you, more specifically we will be exploring the tool vcluster and how to use to create virtual clusters inside of an EKS cluster on AWS.

Requirements to follow along with this article:

• Terraform → https://developer.hashicorp.com/terraform/install

• Helm → https://helm.sh/docs/intro/install/

• vcluster → https://www.vcluster.com/docs/getting-started/setup

• AWS Account

Note that this does exceed the AWS free-tier and can occur charges on your AWS Account depending on how long you keep this setup up and running. Remember to always clean up (terraform destroy) after you are done with your experiments.

Find the necessary code for this exact setup here:
https://github.com/infrastructure-as-posts/vcluster-on-eks

For readability sake the code snippets in the article have been kept short and may be missing some parts, which can be found in the repository linked above.

Why virtual clusters?

Virtual clusters create an appealing middle ground between spinning up multiple real clusters and simply using separate namespaces. They are only slightly more costly than namespaces but create strongly isolated environments.

Think for example about B2B SaaS companies that might currently be spinning up a new Kubernetes cluster for each of their enterprise customers, this comes at a cost and using virtual clusters instead could lead to significant savings for them.

I can also see this being utilized for test environments, where a company might spin up and down virtual Kubernetes clusters for test environments on-demand.

How does it work?

Virtual clusters essentially turn a namespace into a cluster of it’s own. vcluster replicates the components of a standard Kubernetes cluster, such as the API server, controller manager, and scheduler. However, these components interact with the physical cluster's resources in a way that is abstracted and isolated, allowing multiple virtual clusters to coexist on the same physical infrastructure without interference, each believing it's an independent Kubernetes cluster.

As you can see, the pods exist on the Host cluster but everything else, such as

• deployments

• statefulsets

• secrets

• configmaps

• …

Only exists in the virtual cluster and can’t be seen from the host cluster.

Now that you know what we are talking about, let’s setup a new virtual cluster on top of AWS EKS.

Creating a new EKS Cluster

The first thing we need for our setup is a Kubernetes cluster to act as the host cluster. We will be using the Elastic Kubernetes Services (EKS) by AWS. To setup an EKS cluster with Terraform there are two common approaches:

1. eks module provided by the community

2. Official aws_eks_cluster resource

At the time of writing this, the community module has over 1.2 Million downloads in the last month and is one of the most established and well maintained community modules. It makes the whole setup a lot easier and thus we will be using it here as well. If you need more fine grained control you can opt for the aws_eks_cluster resource instead.

The Terraform sample below shows that we will be creating a new VPC for the cluster and that we will have two node groups with one node in each, resulting in a cluster of two nodes in total.

locals {
  cluster_name = "vcluster-eks"
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.0.0"

  name = "vcluster-vpc"

  cidr            = "10.0.0.0/16"
  azs             = ["eu-central-1a", "eu-central-1b"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "19.15.3"

  cluster_name    = local.cluster_name
  cluster_version = "1.28"

  vpc_id                         = module.vpc.vpc_id
  subnet_ids                     = module.vpc.private_subnets
  cluster_endpoint_public_access = true

  eks_managed_node_groups = {
    one = {
      name = "node-group-1"

      instance_types = ["t3.small"]

      min_size     = 1
      max_size     = 1
      desired_size = 1
    }
    two = {
      name = "node-group-2"

      instance_types = ["t3.small"]

      min_size     = 1
      max_size     = 1
      desired_size = 1
    }
  }
}

The following diagram should help in visualizing the different encapsulation layers. The EKS cluster only exists within the boarders of the VPC, it spawns across two availability zones (eu-central-1 and eu-central-1b) and each node of this cluster will be in it’s own subnet within one of the two AZs.

Note that it is not always a given that each node will have their own subnet, if you have many nodes then many of them will also end up in the same AZs and in the same subnet.

Creating a virtual cluster inside EKS

Now that we have an EKS cluster up and running, adding vcluster to it can easily be done via a Helm chart provided by loft labs (the company behind vcluster).

resource "helm_release" "demo_vcluster" {
  name             = "demo-vcluster"
  namespace        = "vcluster"
  create_namespace = true
  repository       = "https://charts.loft.sh"
  chart            = "vcluster"
  version          = "0.18.1"
}

Note that depending on the time you are trying to replicated this setup you might have to change the version number as development around vcluster is happening rapidly and 0.19.0 is already in alpha.

Connecting to the virtual cluster

Connecting to the vcluster is straight forward, let’s first look at the available virtual clusters:

vcluster list

You should see the “demo-vcluster” up and running. The output here is very similar to the one of e.g. ‘docker ps’.

Connecting to the vcluster once it has been created is also straightforward, all you need is the name of virtual cluster.

vcluster connect demo-vcluster

While you are connect, you kube-context has been modified and all “kubectl” commands you run from here on will be targeting the virtual cluster instead of the host cluster.

You can verify that your context has changed by running the command below

$ kubectl config current-context
vcluster_demo-vcluster_vcluster_arn:aws:eks:eu-central-1:345411212130:cluster/vcluster-eks

After connecting to the virtual cluster we can interact with it in the same way we would do with any other Kubernetes cluster. The fact that this is a virtual cluster is completely hidden from us.

Looking at the cluster in k9s we only see a single pod running CoreDNS

Conclusion

Currently virtual clusters seem like a promising technology to me and I could see it being heavily utilized for testing in the future, especially in the context of CI/CD pipelines. They allow for isolated testing environments that closely mimic production settings, enabling more accurate testing and faster deployment cycles. Furthermore, virtual Kubernetes clusters contribute to efficient resource utilization, reducing the cost associated with physical infrastructure.

Closing words

I hope you got some value out of this and consider leaving your email address so that you don’t miss out on any of my future posts. I am also just a human who makes mistakes, so if you think anything here is wrong, please let me know.
If you want to encourage me to keep doing this you can buy me a coffee.

Let’s first first clarify that this article isn’t talking about all of prime video but only about one tool, that one team at Prime Video uses.

Our Video Quality Analysis (VQA) team at Prime Video already owned a tool for audio/video quality inspection, but we never intended nor designed it to run at high scale

One can assume that the entirety of Prime Video still consists of a microservice architecture, evidence for which can be found in one of their other blog posts, and the article above never suggested otherwise.
One fellow redditor has said it best:

You can make a wall out of bricks, and it can be better using 200 bricks vs 5 big blocks, but not using 400,000 bricks.

Emphasizing that they are still using a microservice architecture for prime video, but for this particular service they had split it up too much and found a bigger service to be more beneficial.

So, let’s take a closer look at why the split into multiple microservices wasn’t the right approach in this case. The following image shows their initial microservice based architecture.

Quoting from the article:

The two most expensive operations in terms of cost were the orchestration workflow and when data passed between distributed components

Now, looking at the diagram above, a high cost for data flow is not all that surprising with the media conversion service uploading everything to an S3 bucket first which is then to be accessed by the detectors running on the step functions.

But another factor that played a huge role in this was the cost of the step functions. I am again quoting from the original article:

Our service performed multiple state transitions for every second of the stream

Currently, the price for AWS step functions sits at $0.025 per 1,000 state transitions. If we assume that by “multiple” they mean 3, which is probably a generous assumption, then streaming for one hour would result in

0.27$ for one hour of streaming. This really shows that, while serverless options are often are great way to get started in developing new services and applications, they can get absurdly expensive under heavy workloads.

With this new approach they did run into a problem tho, they got to a point where they couldn’t add any more detectors to the single EC2 instance - a classic problem when running a monolith. They solved this by creating multiple copies of the services with different detectors running on them and added an orchestration layer on top.

Now, looking at the end result, only two things have really changed.

1. They are now using EC2 based ECS tasks for their computation instead of serverless step functions.

2. They integrated the media conversion directly into the detectors

The first change is cheaper for them because step functions get ridiculously expensive at scale, as we saw in the example calculation.
The second change eliminates the intermediate S3 bucket which was frequently accessed, making it an expensive component as well.

Of course, all of this is easy for me to write now looking at their article with all the benefit of hindsight. Making the decision to re-architect something that is already running in production is never easy and making the call to use less of the prominent microservice based approach and instead couple things closer together must have been even harder. My respect to the engineers who made this decision and congratulations to their achievements. Please go and read their article, it’s much better than this one.

Closing Words

I hope got value out of this and consider leaving your email address so that you don’t miss out on any of my future posts. I am also just a human who makes mistakes, so if you think anything here is wrong, please let me know.
If you want to encourage me to keep doing this you can buy me a coffee.

There are a few commercial solutions that allow you to have metrics, logs and traces all in one place, such as DataDog or NewRelic. When you want to selfhost open-source solutions instead you were stuck with using multiple solutions and swichting between them. You might have been using Prometheus for metrics, Jaegar for Traces and Elasticsearch for logs - as an example.

All of these tools are outstanding in their own right and are great at what they do but there are some nice benefits to having all three (logs, metrics, traces) in one place.

Coincidentally, while I was in the process of writing this article there was also a huge DataDog outage which might make some people evaluate other options - including open-source ones.

So, let’s talk about

• Observability & Monitoring

• SigNoz & OpenTelemetry

• The Community

And don’t forget to leave a star for them on Github, they deserve it.

Disclaimer: Observability and Monitoring are huge topics that won’t ever fit into a single blog post, if you really want to get into it I highly recommend the SRE book.

Observability & Monitoring

Observability is a property of a system, like functionality or testability. A system is considered “observable” if the current state can be estimated by only using information from outputs. These outputs are logs, metrics and traces, also known as the three pillars of observability. An application with great observability makes it easy for teams to analyze what is happening and enables them to quickly resolve the underlying issue.

Monitoring on the other hand is the active act of observing the system. In essence, monitoring technologies, such as application performance monitoring (APM), can tell you if a system is up or down or if there is a problem.

These two things usually go hand-in-hand as having better observability gives you more data to monitor and having great observability without any monitoring also means that you won’t ever know when something is wrong before the users complain.

If you search for these two terms online you will also find different definitions, for example some consider monitoring a subset of observability. As with many terms invented by our industry it’s hard to find a clear answer. In the end it will only matter that you know when something is wrong with your system and that you are able to resolve the issue.

When setting up monitoring for any system we generally want it to help us answer the questions: what’s broken, and why?

Note that these two questions often have totally different answers, what’s broken might be an application serving error 500s to users while the why might the server our application is running on is out of space or that a database might be refusing to connect.

What information do we need to answer these two questions? While you can monitor virtually anything about a system and sometimes it has to be decided on a case by case basis what is relevant information for a given system, the industry has generally come to agree on four factors that we should always have a close eye on - the four golden signals:

• Latency

• Traffic

• Errors

• Saturation

If you measure all four golden signals and send out an alert when one of them is problematic, your service will have at least somewhat decent monitoring.

How it works with SigNoz & OpenTelemetry

SigNoz relies on OpenTelemetry to gain insights into the performance of your applications. Once you have instrumented your application with OpenTelemetry, you send this data to the SigNoz Otel Collector, which will save it in a ClickHouse database.

This is easy enough if you develop your application with OpenTelemetry right from the beginning and use it for all signals. When it comes to logging tho, most users will probably already be using a different library.

In those cases, it’s still easy to get your logs into SigNoz you can write your logs to a file and use the OpenTelemetry Filelog Receiver.

If advanced parsing and collecting capabilities are needed which are not present in OpenTelemetry or something like FluentBit or LogStash is already present then the agents can push the logs to the OpenTelemetry collector using protocols like FluentForward, TCP or UDP.

With the data in ClickHouse you can use the SigNoz web interface to gain insights into your applications state and performance.

I also love how sane their default configuration is. When we decided to deploy SigNoz into one of our test environments at work we simply added their helm chart to our Terraform repository and without setting any configuration options it just worked - see the code we used below.

resource "helm_release" "signoz" {
  name             = "signoz"
  repository       = "https://charts.signoz.io"
  chart            = "signoz"
  create_namespace = true
  namespace        = var.namespace
}

I was also amazed that I installed SigNoz in our k8s cluster and it was able to start ingesting k8s pod logs and metrics right away without requiring any further configuration.

The maintainers of SigNoz also provide guidance in how to setup OpenTelemetry and SigNoz for all kinds of different applications

• Flask Python

• NodeJS

• Tomcat Java

• Gin Go
  

And more - If you prefer videos over writing then they also have a series of on youtube on how to setup different their monitoring system for different applications, you can find it here.

The Community

What really made me fall in love with this project is how they treat their community. When discovering a new open-source software that interests me I generally start on the “issues” tab - this gives me a good idea of what’s going on in the project, how active the developers are and how they’re dealing with bugs and feature requests. SigNoz has to be one of the most active projects I’ve discovered so far - turning on notifications means that I can’t open Github anymore without seeing new discussions or pull requests.

You can also see the number of commits to the repository over time - these guys are killing it.

I’ve also Made a few small contributions myself (here, here and here) and felt very welcomed by the developers - looking forward to contribute more in the future.

If you’ve always wanted to contribute to open-source there are a lot of open issues with the “good first issue” label that you could pick up - if anything is unclear just ask and I’m sure they’ll be more than happy to help.

In addition to their activity on Github they are also happy to help anyone troublehsoot problems and discuss ideas on their slack channel.

Conclusion

So, why should you give SigNoz a chance?

• Open-source

• Awesome default settings

• Very easy to setup and get started

• Tries to do lot’s of common things - such as calculating p99 - automatically for you

• Active community and developers who are willing to help you troubleshoot and fix things

All that being said, don’t exepect it to be as mature as DataDog just yet. They are still at an early stage with very active development so expect things to change / break when updating.

I am looking forward to see them develop this further and believe that they can give many of the commercial tools a run for their money.

Closing Words

I hope got value out of this and consider leaving your email address so that you don’t miss out on any of my future posts.

This newsletter is free but if you want to encourage me to keep doing this you can buy me a coffee.

I also hate it whenever I have to wait for a pipeline to finish. Eagerly wanting to know if my change is good and the pipeline passes or if there are problems I need to fix first. Depending on your pipeline this can take anywhere from 1 to 60 minutes. 1 Minute is a pipeline that just runs your unit tests and reports back if they pass or fail. A pipeline that is closer to 60 minutes is doing a whole lot more (at least I would hope so), static code analysis, unit tests, integration tests, linting, deployments to test environments, end to end tests - you can add a lot to your pipeline if you so choose.

However long your pipeline currently takes, you would probably prefer it to be faster and conveniently today I want to talk about a few things that may help you get there.

I have mostly been using Gitlab pipelines in my own career and I will be using Gitlab for all examples in this Post. That being said, most principles you find here can also be implemented in other CI/CD platforms like CircleCI or Github Actions.

Before we begin I want to at least establish some common vocabulary.

• Runner → Maschine that executes the pipeline

• CICD Platform → Where we define pipeline instructions, e. g. Gitlab

• Job → A group of instructions that is executed by the runner

• Stage →A group of one or more jobs, common stages are: build, test, deploy

The things I will be talking about here are:

• When to run pipelines

• Caching

• Docker in Docker

All I say here stems from personal experience and your mileage may warry.

When to run pipelines

Using one repository for multiple applications seems to have become a popular approach in recent years, at least I have stumbled upon it on multiple occasions. I first hard about Google doing it and later discovered that there is even a word for it, Monorepo.

I have also seen some strange patterns emerge when following the monorepo approach, such as running all tests for all services whenever a change has been made to any service.

Always testing all applications in a repo with potentially 10s, 100s or even 1000s of them can be unfathomably time consuming. If you are using cloud provider to run your Pipelines, you might also have to pay more for the heavier load.

Just because all your applications and services are now living in one repository doesn’t mean that you should treat them like one single entity. In the world of microservices such an approach would be called a distributed monolith.

Many CI/CD platforms offer a way to check for changes to certain files or directories before running a pipeline. In Gitlab you can do it as follows:

only:
  changes:
    - dir-a/*
    - dir-b/**/*
    - file-c

This defines a pipeline that is only executed when

a) a file in dir-a has been changed
b) a file in dir-b or any subdirectory of dir-b has been changed
c) file-c has been changed.

In a monorepo this can be used to run the tests for a service only when files for this service change.

Builds and deployments should generally be handled similarly. They are all different services that just happen to have their code in the same repository, treat them as different entities.

Be aware that `changes` in Gitlab is always considered true when the pipeline is triggered via the web UI instead of by pushing code.

Another case I come across frequently is to manually trigger only certain parts of the pipeline and all CI/CD platforms should allow for this in one way or another. In Gitlab you can define custom variables before triggering a Pipeline manually and then check these variables for running certain jobs.

For example, if I have two separate test jobs defined, TestA and TestB then I can define that TestA may only run if the variable TEST_A is set to yes

only:
  variables: 
    - $TEST_A == "yes"

When you define many rules around when to run your pipelines, things can get a bit confusing, for example consider the following `only` definition. This Pipeline will run when either TEST_A or TEST_C is yes.

only:
  variables: 
    - $TEST_A == "yes"
    - $TEST_C == "yes"

But what about the following?

only:
  variables: 
    - $TEST_A == "yes"
    - $TEST_C == "yes"
  changes:
    - dir-a/*
    - dir-b/**/*
    - file-c

Well this will also run when the Pipeline is triggered manually and either TEST_A or TEST_C is set to yes. It will never run on any code push tho, because both the variables and the changes section have to evaluate to true. When you trigger the Pipeline manually, changes are always true, so in that case it works, but if you push code changes you never have custom variables defined thus your pipeline will never run.

More recently, Gitlab has introduced a new alternative to only in the form of rules. With rules I was able to get the behavior I actually desired, the Pipeline can be run manually by setting either TEST_A or TEST_B and the Pipeline will also automatically run all tests on a push that contains changes to the code.

rules:
  - if: '$TEST_A == "yes" || $TEST_ALL == "yes"'
  - if: '$TEST_B == "yes"
    when: never
  - changes:
      paths:
        - dir-a/**/*
        - dir-b/**/*

Caching

Let’s say we have an application that is build using nodejs. Installing all dependencies for a big javascript application can take a while, so long in fact that the following has become one of the most wide spread memes among developers.

Which leads us to the conclusion, that we want to avoid downloading dependencies whenever possible.

How can we determine when the node_modules should be downloaded again? Pause here for a moment and think about it, it’s a good mental exercise.

The answer is to take a hash of the package-lock.json file. When this hash changes, the dependencies have changed in some way and need to be downloaded again. As long as this hash stays the same, we keep using the present node_modules and skip the download step.

Gitlab will do this hashing step for us if we us package-lock.json as the cache-key.

cache:  
  key:    
    files:      
      - package-lock.json  
  paths:    
    - node_modules/

Using distributed caching

When you have one more than one runner for your pipeline, caching on the runner becomes less effective. You might have 6 jobs that run in parallel and that all need to be done before the next stage. If only one of these 6 jobs get picked up by a runner that doesn’t have his cache build up, this one becomes the bottleneck it almost doesn’t matter that we had a cache hit on the other 5.

In that case, it can make sense to store your cache in a remote storage such as S3 that all runners can access. Downloading your node_modules from S3 will of course be slower than having them cached locally but in my experience it is still a lot faster than running npm install.

Docker in Docker

Do you really want Docker-in-Docker? Or do you just want to be able to run Docker (specifically: build, run, sometimes push containers and images) from your CI system, while this CI system itself is in a container?

The above is a quote from this excellent article by one of the developers of docker, who has also worked on docker in docker. Since we are talking about pipelines here, we want the later. I strongly advise you to read their whole article but If you are short on time, you should mount the docker.sock of your runner into the container that executes the pipeline.

With this, when you use a docker command inside the docker runner, the command will be executed by the host docker engine. Meaning that instead of running docker containers inside docker containers you instead get the host to create a second container.

In Gitlab, you would do this by modifying the runner configuration as seen below.

[[runners]]
  url = "https://gitlab.com/"
  token = RUNNER_TOKEN
  executor = "docker"
  [runners.docker]
    image = "docker:20.10.16"
    privileged = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]

Lessons learned

The most important take aways here are

• Only test / build / deploy what has changed

• Make use of caching mechanisms

• Docker in Docker is often not necessary for CI purposes

Closing Words

I hope got value out of this and consider leaving your email address so that you don’t miss out on any of my future posts.

Also, we are hiring - so if you are you are in Germany (Hamburg) or willing to move here and you would like to join me, we are looking for DevOps folks as well as Go developers. Get in touch with me on LinkedIn if you wana know more.

This newsletter is free but if you want to encourage me to keep doing this you can buy me a coffee.

When people think about ingresses on Kubernetes, they are commonly imagining them as the entrypoint into the cluster that also lives inside the cluster. Often represented in diagramms such as the one below.

Here the traffic arrvies in our cluster, the ingress forwards it to a service of type ClusterIP which then hands it of to the pods. The service can be of type ClusterIP since it doesn’t have to be accessible from outside the cluster, it’s only accessed via an Ingress that lives inside the cluster.

I also always thought of them like this, which stems from a youtube video I watched early on when I started learning about kubernetes.

This is not necessarily wrong but as it is often the case with diagramms, they don’t tell you the whole story.

An ingress is really just a set of rules to pass to a controller that is listening for them. An Ingress Controller. You can deploy a bunch of ingress rules, but nothing will happen unless you have a controller that can process them.

When you set up the AWS Load Balancer Controller as an Ingress Controller, they tell you that the service must be of type NodePort or LoadBalancer. At least when using their instance mode.

service must be of type "NodePort" or "LoadBalancer" to use instance mode

I don’t wanna go into the details about the two different modes or tell you which one you should choose, at least not in this article.

I was wondering why these services would have to be of type "NodePort" or "LoadBalancer". Setting them up as such makes them reachable from outside the cluster - but when we have an ingress in front, why would this be necessary?

I finally understood when looking at an architecture diagramm of the AWS Load Balancer Controller.

When you are using AWS Load Balancer Controller as your ingress controller, the ingress rules are fulllfilled by target groups in AWS, outside of your cluster.

The diagram from the beginning of the post would have to look more like the one below.

Here you can see that the service has to be reachable from outside the cluster, which means that we can’t use type ClusterIP but we have to use NodePort or LoadBalancer.

Lessons learned

An Ingress is just a set of rules and it’s up to the ingress controller to enforce these rules. In most cases, the first diagramm we saw will probably be correct and your ingress will be part of your cluster but it doesn’t have to be that way.
The AWS Load Balancer Controller is using AWS Target Groups to enforce the ingress rules before traffic enters your kubernetes cluster.

Closing Words

I hope got value out of this and consider leaving your email address so that you don’t miss out on any of my future posts.

This newsletter is free but if you want to encourage me to keep doing this you can buy me a coffee or buy me a coffee.

Let me start by giving you some background knowledge on the two components that were involved in this case, Strimzi Kafka Operator and AWS Load Balancer Controller.

Kafka on Kubernetes

We are using kafka at work, deployed on kubernetes using the Strimzi kafka operator.
Furthermore, we are exposing kafka using load balancers, as has been showcased on the Strimzi blog.

To give clients access to the individual brokers, the operator creates a separate service of type LoadBalancer for each broker. As a result, each broker will get also have a separate load balancer in the cloud.

This leads to a slightly odd architecture, as shown in the diagramm below, assuming you have 3 brokers in your kafka cluster.

None of these load balancers are actually balancing any load, they just serve as entrypoints for each individual kafka broker.

It is also worth mentioning that these load balancers on AWS are Network Load Balancer (NLB). Network load balancers act on layer 4, this enables them to serve traffic with a lower latency and at less cost than the layer Application Load Balancer (ALB) but they also don’t offer features such as authentication, sticky sessions or request based routing.

We can find a simple answer as to why they went with NLBs over ALBs on the strimzi blog

Since none of the common load balancing services supports the Kafka protocol, Strimzi always uses the Layer 4 load balancing.

AWS Load Balancer Controller

As you saw in the last image, for each broker we need a service of type LoadBalancer. We are running our entire infrastructure on AWS, so for us each of these services maps to an external load balancer in the cloud, an ELB.

These load balancers on AWS are automatically created for us by the AWS Load Balancer Controller. This is generally great as we don’t have to worry about managing these load balancers, we just create a service of type LoadBalancer in Kubernetes and the controller takes care of creating and managing the ELB automatically.

It also takes care of deleting the load balancer on AWS once we delete the service in our cluster. To achieve this it adds a finalizer to the definition of all services (and ingresses) that it created a cloud resource for.

For an explanation of how finalizers work have a look at last weeks episode or the finalizers documentation.

Mistery of a missing finalizer

We noticed that when removing the LoadBalancer services that connect to Kafka in our cluster, the ELBs on the side of AWS are still there even tho they don’t serve a purpose anymore.

First I checked the current yaml definition of our load balancers, created by the strimzi operator. There were no finalizers, which is clearly why the load balancers in AWS didn’t get deleted. Which leaves only one question to be answered, why are the finalizers missing?

During research I found this issue in the strimzi operator repo according to which support for finalizers should have been added around march 2021. Furthermore a second issue in the same repository suggest adding the following.

finalizers:
- service.kubernetes.io/load-balancer-cleanup

I had to look up what this is supposedly doing and found a straight forward explanation.

Specifically, if a Service has type LoadBalancer, the service controller will attach a finalizer named service.kubernetes.io/load-balancer-cleanup. The finalizer will only be removed after the load balancer resource is cleaned up. This prevents dangling load balancer resources even in corner cases such as the service controller crashing.

Now, in hindsight, I should have paused here for a moment and ask myself why the service controller, who is supposed to add this finalizer, apparently didn’t do it. That’s not what I did tho, instead I rushed towards my editor and added this finalizer myself.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: {{ .Values.cluster_name }}
  namespace: {{ .Values.namespace }}
spec:
  kafka:
    version: {{ .Values.kafka_version }}
    replicas: {{ .Values.kafka_replicas }}
    authorization:
      type: simple
      superUsers:
        - admin
    listeners:
      - name: plain
        port: 9095
        type: internal
        tls: false
        authentication:
          type: scram-sha-512
        configuration:
          useServiceDnsDomain: true
      - name: external
        port: {{ .Values.kafka_port }}
        type: loadbalancer
        tls: true
        authentication:
          type: scram-sha-512
        configuration:
          finalizers:
            - service.kubernetes.io/load-balancer-cleanup
        ...

With this change deployed in one of our test environments I used k9s to confirm the presence of this finalizer.

What I saw really suprised me tho, there are suddenly not just one but two finalizers present.

• service.kubernetes.io/load-balancer-cleanup

• service.k8s.aws/resources

At that time I didn’t know where this second finalizer came from and felt super confused.

Things got worse when I tried to delete this service and saw it being removed instantly. Normally when a finalizer is present, this should take a few seconds for the controller to remove the cloud resource.

I went over to the AWS console and, of course, the load balancer is still there. The controller did not remove it even tho the service of type LoadBalancer was deleted. The added finalizer did not work.

I had no idea what was going on at this point but this other finalizer that had appeared, service.k8s.aws/resources, seemed to be worth investigating.

Great, the first hit is a code reference on Github, I already thought this would be a deep rabbithole to go down. Then I noticed the name of the rep aws-load-balancer-controller - so this other finalizer had to be related to it.

Then it dawned on me, the operator first creates the service of type LoadBalancer, then the load balancer controller adds the finalizer (service.k8s.aws/resources) to it and so far everthing is good but then the operator removes the finalizer in his reconciliation loop.

When I saw both finalizers being present, the operator must have removed the service.k8s.aws/resources just before I could delete the service.

In this case when we remove the service type LoadBalancer, the ELB we will not be deleted. Without the finalizer the service gets deleted immediately and the AWS Load Balancer Controller has no idea that it has to get rid of the cloud load balancer.

When we now go back to the defintion file and replace service.kubernetes.io/load-balancer-cleanup with service.k8s.aws/resources then the whole things works as intended.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: {{ .Values.cluster_name }}
  namespace: {{ .Values.namespace }}
spec:
  kafka:
    version: {{ .Values.kafka_version }}
    replicas: {{ .Values.kafka_replicas }}
    authorization:
      type: simple
      superUsers:
        - admin
    listeners:
      - name: plain
        port: 9095
        type: internal
        tls: false
        authentication:
          type: scram-sha-512
        configuration:
          useServiceDnsDomain: true
      - name: external
        port: {{ .Values.kafka_port }}
        type: loadbalancer
        tls: true
        authentication:
          type: scram-sha-512
        configuration:
          finalizers:
            - service.k8s.aws/resources

Now there is only one finalizer present and it’s the one that the AWS Load Balancer Controller understands.

Deleting the service again will take a few seconds, as it should, and leaves no cloud load balancer behind.

Lessions learned

1. Finalizers are just a string that your controller understands. Adding the finalizer service.kubernetes.io/load-balancer-cleanup didn’t help because it’s not what the AWS Load Balancer Controller was expecting.

2. When something is missing from or unexpectedly added to a service defintion and the service is handled by an operator, there is a good chance that the operator is to blame. Keep the reconciliation loop in mind when troubleshooting such cases.

Closing Words

In this post I tried to share the whole process of fixing this issue, including all the gaps in my knowledge that I had and how I closed them.

I hope you enjoyed this little story and that you will be better equiped should you ever face a similar situation. If you got value out of this then consider leaving your email address so that you don’t miss out on any of my future posts.

This newsletter is free but if you want to encourage me to keep doing this you can buy me a coffee or buy me a coffee.

We will also look take a brief look at the concept of finalizers and see how they help us avoid leaving unwanted cloud resources activ when deleting load balancers in Kubernetes.

Services in Kubernetes

Each Pod in Kubernetes has it’s own IP address. When a Pod is destroyed and recreated by Kubernetes, it’s IP changes. This is obviously a problem if we want to access an application running on a Pod. Services solve this problem for us by giving us a static IP through which we can access an application running on a Pod.

The default service type is ClusterIP. This type of service can only be accessed from inside the cluster. Services of type NodePort or LoadBalancer are accessible from outside the cluster.

Service type LoadBalancer vs Ingress

Kubernetes services of type LoadBalancer and Ingresses can at first glance seem like they are doing the same thing. So let’s take a closer look.

Service type LoadBalancer

A kubernetes LoadBalancer service is a service that can be accessed through external load balancers that are NOT in your kubernetes cluster, but exist elsewhere. They can work with your pods, assuming that your pods are externally routable. Google and AWS provide this capability natively. In terms of Amazon, this service maps directly with ELB. Kubernetes when running in AWS (EKS) can automatically provision and configure an ELB instance for each LoadBalancer service deployed.

The following shows how users can access pods and the applications running on them, on kubernetes when using a service type LoadBalancer with AWS.

When you are running kubernetes on premise you might not want to depend on a cloud provider for load balancing. In that case there are also alternatives such as metallb.

Ingress

An ingress is really just a set of rules to pass to a controller that is listening for them. An Ingress Controller. You can deploy a bunch of ingress rules, but nothing will happen unless you have a controller that can process them. A LoadBalancer service could listen for ingress rules, if it is configured to do so.

Let\s look at an example ingress ressource

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-example
spec:
  rules:
  - host: "foo.bar.com"
    http:
      paths:
      - pathType: Prefix
        path: "/bar"
        backend:
          service:
            name: service1
            port:
              number: 80
  - host: "bar.foo.com"
    http:
      paths:
      - pathType: Prefix
        path: "/foo"
        backend:
          service:
            name: service2
            port:
              number: 80

Each ingress rule consists of the following:

• Host

• Paths

• Backend

The host is optional, in this example, the host is specified for both ingress rules, so the rule applies to inbound HTTP traffic where the host header is either foo.bar.com or bar.foo.com.

The paths section contains a list of paths, in the example above each ingress rule has one path in it’s list of paths. Each path is associated with a backend

The backend defines a service to which the traffic should be send. HTTP(S) requests that match the host and path of the rule are sent to the listed backend.

Let’s consider a second ingress resource and a visual representation of it.
This time our ingress consists of a total of three rules, all applying to the same host example.com. They differ by path, meaning that an Incoming request which has the host header set to example.com and a path of s1 will be forwarded to service1.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-example
spec:
  rules:
  - host: "example.com"
    http:
      paths:
      - pathType: Prefix
        path: "/s1"
        backend:
          service:
            name: service1
            port:
              number: 80
  - host: "example.com"
    http:
      paths:
      - pathType: Prefix
        path: "/s2"
        backend:
          service:
            name: service2
            port:
              number: 80
  - host: "example.com"
    http:
      paths:
      - pathType: Prefix
        path: "/s3"
        backend:
          service:
            name: service3
            port:
              number: 80

The following diagramm is a visual representation of the above ingress.

Service1-3 in the above example are typically of type ClusterIP since they are accessed from inside the cluster.

As you can see, the ingress rules define which service the incoming traffic should go to. It’s important to once more emphasize that ingress rules alone have no effect, you need a controller that executes them.

While a service of type LoadBalancer or Nodeport may be used with any portocol or port, an Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type NodePort or LoadBalancer.

When to use what?

If you want to directly expose a service, service type LoadBalancer is the default method. All traffic on the port you specify will be forwarded to the service. There is no filtering, no routing, etc. This means you can send almost any kind of traffic to it, like HTTP, TCP, UDP, Websockets, gRPC, or whatever.

The big downside is that each service you expose with a LoadBalancer will get its own IP address, and you have to pay for a LoadBalancer per exposed service, which can get expensive!

Ingress is probably the most powerful way to expose your services, but can also be the most complicated. There are many types of Ingress controllers, from the Google Cloud Load Balancer, Nginx, Contour, Istio, and more. There are also plugins for Ingress controllers, like the cert-manager, that can automatically provision SSL certificates for your services.

Ingress is the most useful if you want to expose multiple services under the same IP address, and these services all use the same L7 protocol (typically HTTP). You only pay for one load balancer if you are using the native GCP integration, and because Ingress is “smart” you can get a lot of features out of the box (like SSL, Auth, Routing, etc)

AWS Load Balancer Controller

Now, as we have seen before, the service type LoadBalance needs an external load balancer to function. In most cases these will be load balancers created on a cloud provider.

The exact setup of this varies depending on the cloud provider you use, assuming AWS you will need the AWS Load Balancer Controller.

This controller coordinates the creation of an ELB on AWS whenever you create a service of type LoadBalancer in your kubernetes cluster.

When you create a service of type LoadBalancer the AWS Load Balancer Controller will communicate with AWS to ensure that the corelating ELB is created there.

Finalizers for load balancers

Now, what happens when we delete the serivce type LoadBalancer? Ideally we want the actual load balancer in the cloud to be gone as well since it’s serves no purpose without the service.

Good news: that’s exactly what happens. when you have the AWS Load Balancer Controller deployed on your kubernetes cluster, you might notice that all your services of type LoadBalancer have a section added to their metadata that looks like this:

finalizer:
  - service.k8s.aws/resources

Similarly, all ingresses in your cluster also have a finalizer section

finalizer:
  - ingress.k8s.aws/resources

These finalizers will prevent the resource from being deleted until the AWS Load Balancer Controller is done cleaning up all related cloud resources.

The name of the finalizer, e.g. `service.k8s.aws/resources` really has no special meaning to it in kubernetes, it’s just a string that the load balancer controller understands.

If you put a finalizer such as

finalizer:
  - this.is.just.a.random.string/doesnotexist

Then the deletion of the resource will be prevented indefinetly, since there is no controller that understands this strings and knows what to do with it. The only way you can delete the service in this case to manually remove the finalizer from the service definition.

Closing Words

Next Weeks episode will feature a story about how AWS Load Balancer Controller and Strimzi Kafka Operator can be working against each other, troubleshooting an edge case that almost had me lose my sanity. Consider leaving your email if you want to hear the story.

Openlens and k9s offer visibility into your Kubernetes cluster. Running a kubectl command everytime you need any information about your cluster can be cumbersome. The obvious solution to this is to have some kind of UI that shows you all the resources in your cluster.

As many people felt the need to have a UI for their Kubernetes clusters, many different solutions have emerged. The most prominent one are:

• kube-dashboard

• Lens

• k9s

• kubenav (mobile)

And there are even more than just these 4.

Lens is an electron app with both a commerical version (called Lens) and a free and open source version (called Openlens). I have been a long time user of openlens, mostly because it's what my company was already using when I joined. The free version has been working just fine for me and had everything I could ask for, until it didn't. Recently, the lens developers have decided to downgrade Openlens in order to get more people to use their commerical version.

Many users where caught by suprise when they updated Openlens to version 6.3.0 and discovered that the buttons for accessing logs and getting a shell on a pod where gone. In a discussion on a Github issue, opened by one such surprised user, they proclaimed to have made these changes for “more secure and faster booting” while at the same time pointing out that these buttons are still available in the commercial version.

Not sure when the last time was that I saw this many downvotes on Github.

They surely got a lot of negative feedback, people are especially furious about the fact that they tried to sell this as "for security reasons". People don't like it when you lie to their face, who would have thought.

I liked Oenlens but now, the free version is not up to the task anymore and I don't want to support a company that downgrades an open source product and lies to their uses by calling it a security patch.

Anyway, there were many people talking (complaining) about this change on both Reddit and Twitter. The most recommended alternative in all of these discussion was k9s which I first tried about two weeks ago and so far I have been loving it.

There is a clear downside to it tho, it’s a terminal UI. This means no more clicking around, you will need to know keyboard shortcuts to get things done.

The screenshot below show k9s having me choose between the clusters configured in my ~/.kube/config file

Once you get used to to this way of navigating your cluster, it's just so much faster. Even if the Lens developers rolled back their changes and added the buttons back in, I wouldn't return.

Anyone who likes to stay in the terminal during development (I mean you, vim users), will love this.

Working with k9s

Follow the instructions here to install k9s.

Once you have it installed, the command k9s will bring up the terminal UI.

If you have multiple clusters in you kubeconfig, the first screen will generally prompt you to choose a cluster, as seen in the previous screenshot.

You navigate up and down this list with j and k, which you might be familar with from the editor vim

Pressing : will bring up a prompt for you to enter commands. Some of the most common comands are

• ctx → brings up the screen to choose a cluster again

• ctx SANDBOX → will directly bring you to the cluster called SANDBOX

• ns → brings up a screen where you switch between namespaces

• pod → will you show you all pods in the current namespace, the same goes for the name of all other k8s ressources, e.g. service will bring up all services

Some other good to know shortcuts

• s while hovering over a pod will give you a shell (ctrl d will let you leave the shell again and return to k9s)

• l while hovering over a pod will give you logs (esc to go back, this goes for everything except the shell)

• ctrl-k will kill a pod

• ctrl-d will delete a pod

• y will give you the yaml definition of a pod

• shift-f (F) to setup port forwarding

Many useful shortcuts are also displayed on the top of the UI. There you can also always see which cluster you are operating in.

Namespaces you visit will also get assigned to numbers. In the screenshot below I`ve been visiting signoz and all namespaces and can now quickly switch between them by pressing 0 for all or 1 for signoz

Typing ? anywhere will bring up the help menu which shows you all shortcuts available

Conclusion

After 6.3.0 openlens can no longer really be considered a good option for visibility into your Kubernetes clusters. For companys that don’t mind paying for the commercial version of lens, it’s still a good opiton. That being said, k9s offers all the features of you need to operate your clusters effectively and will most likely stay free forever. When you work with it regularly and get used to the shortcuts, there is a good chance you will find yourself being faster then you could ever be with lens.

This newsletter is free but if you want to you can buy me a coffee / buy me a coffee.