For little over a year now, zvelo has been transitioning our infrastructure to the Kubernetes Platform. Developed by Google, Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. Using Kubernetes allows us extreme flexibility with our applications, both horizontally and vertically.
In this blog entry, I will share how we arrived at the decision to move to production grade container orchestration and some details around the challenges we faced along our journey. In second part of this blog series, we’ll share some recommendations on how you can best manage Kubernetes with kops on AWS.
What Are We Moving To Kubernetes and Why?
Before transitioning to this new platform, we were using the standard data center-centric infrastructure. While this had worked well for us previously, we were now approaching a critical point due to the increasing demand and growth of our applications/customers/business and this situation necessitated that we transition to another framework. The cost to maintain a standard data center-centric infrastructure would only continue to rise over time and we needed to pivot to something that could support our applications into the future. Over the course of year while diving into the ideas containerization and the future of infrastructure, we have discovered and fully embraced Kubernetes.
Before we could learn how to use Kubernetes to automate deployments of new code, scale cluster capacity up and down, and manage the configuration of containerized applications, we needed to decide where to host our cluster. By analyzing costs, considering add-ons, and additional factors we agreed to go with Amazon Web Services (AWS).
Next, in order to successfully create our Kubernetes cluster in AWS, we worked with a wide variety of available options – everything from kube-up and kube-aws to a set of custom Terraform scripts built from kube-aws.
Of these options, we found that using Terraform awarded us the most flexibility and enabled us to tie in useful infrastructure into our clusters – such as elasticache, redshift, kinesis, and also a variety of IAM roles, policies and permissions to support our application layer. Also, by using this powerful tool , we realized other clear advantages such as bringing clusters down were effortless and making iterating through cluster setups also relatively painless.
Why We Chose To Leverage kops as a Tool?
As new features were added into Kubernetes, maintaining Terraform scripts became increasingly difficult. For example, bootstrapping a node in the cluster while becoming more stable and resilient, was was also becoming non-trivial.
With such an investment already in terraform, we sought a tool to address our challenges. We found Kops which is not only a tool for launching Kubernetes clusters, but also a tool for managing the lifecycle of those clusters. Kops knows how use Terraform with the option ‘–target=terraform’, so we could rely on kops to manage the hard part of keeping Kubernetes up to date, while still being able to utilize our Terraform config files that add all the other AWS goodies that we depend on.
Our method was to use ‘–target=terraform’ to export our config to Terraform files, and then edit them. We soon realized is that it became cumbersome to manage any updates to the cluster provided through kops. While it was obvious that we needed to fully embrace kops, we could not do that while still trying to build the entire cluster out with Terraform. We needed to let go of Terraform a little to have them play nicely together. We decided to manage our cluster with kops, and our add-on infrastructure with Terraform.
We spent a considerable amount of time experimenting with Kubernetes. During this time, we tried out various open source tools and we even developed our own. We found that using kops and Terraform allows us to quickly and reliably create our Kubernetes clusters within the AWS infrastructure. Using Terraform lead us to kops, however they are not dependant on each other. Both of these tools are excellent at what they do.
In our next installment in this blog series, I’ll explain how we are using kops with AWS.
