Cloud giant Amazon Web Services manages the largest number of Kubernetes clusters in the world, according to the company. In this podcast recording, AWS Senior Engineer Jay Pipes discusses AWS' use of Kubernetes, as well as the company's contribution to the Kubernetes code base. The interview was recorded at KubeCon North America last month. Jay Pipes - @jaypipes Joab Jackson - @Joab_Jackson The New Stack - @thenewstack
Cloud giant Amazon Web Services manages the largest number of Kubernetes clusters in the world, according to the company. In this podcast recording, AWS Senior Engineer Jay Pipes discusses AWS' use of Kubernetes, as well as the company's contribution to the Kubernetes code base. The interview was recorded at KubeCon North America last month.
Kubernetes is an open source container orchestration platform. AWS is one of the largest providers of cloud services. In 2021, the company generated $61.1 billion in revenue, worldwide. AWS provides a commercial Kubernetes service, called the Amazon Elastic Kubernetes Service (EKS). It simplifies the Kubernetes experience by adding a control plane and worker nodes.
In addition to providing a commercial Kubernetes service, AWS supports the development of Kubernetes, by dedicating engineers to the work on the open source project.
"It's a responsibility of all of the engineers in the service team to be aware of what's going on and the upstream community to be contributing to that upstream community, and making it succeed," Pipes said. "If the upstream open source projects upon which we depend are suffering or not doing well, then our service is not going to do well. And by the same token, if we can help that upstream project or project to be successful, that means our service is going to be more successful."
In addition to EKS, AWS has also a number of other tools to help Kubernetes users. One is Karpenter, an open-source, flexible, high-performance Kubernetes cluster autoscaler built with AWS. Karpenter provides more fine-grained scaling capabilities, compared to Kubernetes' built-in Cluster Autoscaler, Pipes said. Instead of using Cluster Autoscaler, Karpenter deploys AWS' own Fleet API, which offers superior scheduling capabilities.
Another tool for Kubernetes users is cdk8s, which is an open-source software development framework for defining Kubernetes applications and reusable abstractions using familiar programming languages and rich object-oriented APIs. It is similar to the AWS Cloud Development Kit (CDK), which helps users deploy applications using AWS CloudFormation, but instead of the output being a CloudFormation template, the output is a YAML manifest that can be understood by Kubernetes.
In addition to providing open source development help to Kubernetes, AWS has offered to help defray the considerable expenses of hosting the Kubernetes development and deployment process. Currently, the Kubernetes upstream build process is hosted on the Google Cloud Platform, and artifact registry is hosted in Google's container registry, and totals about 1.5TB worth of storage. Each month, AWS alone was paying $90-$100,000 a month for egress costs, just to have the Kubernetes code on an AWS-hosted infrastructure, Pipes said.
AWS has been working on a mirror of the Kubernetes assets that would reside on the company's own cloud servers, thereby eliminating the Google egress costs typically borne by the Cloud Native Computing Foundation.
"By doing that we completely eliminate the egress costs out of Google data centers and into AWS data centers," Pipes said.
Alex Williams 0:01
You're listening to the new stack makers, a podcast made for people who develop, deploy and manage at scale software. For more conversations and articles go to the new stack dot I O. All right now on with the show.
Colleen Coll 0:20
Amazon Web Services is the world's most comprehensive and broadly adopted cloud platform, offering over 175 fully featured services from datacenters. Globally, millions of customers trust AWS to power their infrastructure become more agile and lower costs.
Joab Jackson 0:41
Hello, and welcome to the latest edition of the new stack makers Podcast. Today, we'll be talking with Jay pipes, a principal engineer at Amazon Web Services. Originally, we were supposed to record this at cube con, but it didn't work for flight schedule. So what do we get now? But anyway, we're here to hear about how the world's largest cloud service uses Kubernetes. And just as importantly, how they are contributed to Kubernetes and contributing to cloud native open source technologies in general. Jay, welcome to the show.
jay pipes 1:17
Thanks for having me, John. Appreciate it.
Joab Jackson 1:18
Excellent, say works on cloud native technologies in the AWS Kubernetes team, focusing on open source contributions, and the Kubernetes ecosystem. He also leads the AWS controllers for Kubernetes AK project, and he helps to shape the upstream open source contribution strategies of AWS engineers. So let's dig in there. First of all, you're very curious, how big is the AWS Kubernetes? Open Source team?
Jay Pipes 1:51
Well, we don't actually have a separate open source team. So at AWS, we take an opinion that if a service, like Eks is built on top of open source technology, Eks is the elastic Kubernetes service. Yes, thank you. Yeah. So if a service is built on top of open source technology, right, like UCaaS, is built on top of Kubernetes. And, therefore is dependent on the success of that open source project, then it's a responsibility of all of the engineers in the service team to be aware of what's going on in the upstream community to be contributing to that upstream community and making it succeed, right, because we just say, Well, what is the upstream open source projects upon which we depend are suffering or not doing well, then our service is not going to do well, right. And, by the same token, if we can help that upstream project or projects be successful, then our service is going to be more successful. So we don't necessarily have a, you know, a separate open source team that only works in open source, and doesn't work on internal code or anything like that. Rather, we just encourage all of our engineers, regardless of the sub team, that they're on, to contribute in any way that they can to them upstream projects.
Joab Jackson 3:21
Excellent. Excellent. Are there particular aspects of Kubernetes? That AWS kind of specializes in? Or are there problems that you guys have had to solve? Yeah.
Jay Pipes 3:33
For sure, yeah. Eks, I'm pretty sure has the largest number of Kubernetes clusters under management in the world, or at least, I think, and having that many Kubernetes clusters under management means that our service team is exposed to where Kubernetes. And that CD, the underlying data store that Kubernetes uses were where the scaling limitations and the production, operational readiness limitations of that software where the breaking points are, right, like so many of the contributions that our team members do for upstream Kubernetes and SCD, and container D and these other technologies is around making sure that the those constraints the scaling constraints and Production Readiness constraints that we've run into, that we provide insights to the upstream community, hey, this is where things break down. This x x number of notes or X amount of traffic to a particular Kubernetes API server, or, you know, a good example of a contribution that came out of our performance and scalability sub team within Eks recently, we have an engineer Shamji to go to who that look Get the logs and metric collection from 1000s of production Eks clusters determined that the Jesus compression that is enabled inside the Kubernetes API server, ostensibly to, you know, reduce the network bandwidth used and decrease the latency that clients experience when talking to an API server that didn't actually was increasing the latency for large lists requests that the clients were making to the Kubernetes API server. And so did a deep dive into trying to figure out what was it a particular compression level that Jesus was hard coded at for? Like, could we reduce that compression level? Could we disable Jesus compression entirely? what impact that would have on latency and network bandwidth? Those are the kinds of contributions to upstream at CD and core Kubernetes, that you'll see coming out of AWS service teams, because those are the kinds of things that our customers report, say, Oh, we're experiencing the slowdown in the Kubernetes API server when we have, I don't know, 1000 nodes and 200,000 objects of a certain chi, we're seeing like increased latencies, we don't really know what's going on. However, engineers go and diagnose what's going on, and put together a bunch of troubleshooting information and collate that into white papers and proposals to upstream Kubernetes, on how to fix those things. So that's, that's the kind of contribution that we like to spearhead. Right. It's direct feedback that we get from our customers that we sort of translate or convert into feature proposals bug fixes in upstream to address those specific problems that we run into,
Joab Jackson 6:51
you know, the end of the story with the Jesup latency issue. What was Yeah,
Jay Pipes 6:55
so Shawn was very detail oriented, I love it, he put together an upstream proposal that had some short term fixes some sort of medium term fixes and some long term things. They've already merged some of the short term fixes, which is sort of like getting some some flags to disable compression and allow the client to disable compression. There's some medium term fixes that have been merged, I can't quite remember what they are, it's a long term ones are tight, like the API server actually sort of tracking which particular kinds of API requests to enable or disable G set compression for, which makes the most sense and sort of doing sort of learning over time, you know, to be the most efficient for a particular kind of watch, or listen to requests and different types of objects that are represented in in sed under the hood that have different compression ratios, right. And so, for instance, you know, a pod or a secret or config map have very different compression ratios. And turning on compression, turning on sheet compression makes a big difference, or no difference at all for some of those objects. And so sort of the long term solution in this space would be the API server sort of understanding, you know, which kinds of resources that it is serving responses for make sense to enable compression for Nice, nice. Thanks. Just one example of the type of contribution to Kubernetes and SED that we like to focus on. I will say that our number one priority for all of our open source is any integration between the open source project and AWS services. I mean, it's always been that way. It's just it's sort of table stakes for us. So things like the AWS load bouncer controller or the IAM authenticator, right. These are table stakes for us. We prioritize contributions for those AWS specific integrations like cloud provider, AWS, and Kubernetes. Right. These are the things that we absolutely have to prioritize first, because we want Kubernetes to be run on AWS very well, right. We want easy to do to be the best place to run compute containerized workloads on Kubernetes. So it has to focus on those integration pieces like the VPC CNI and load bouncer controller. So that is really where our bread and butter
Joab Jackson 9:27
Nice, nice. So you also in addition to contributing upstream you've also AWS has also released open source software I guess that's come out of this work as well. Yeah, I'm thinking can you talk about carpenter week really cover carpenter and often the new stack?
Jay Pipes 9:44
Yeah, I can certainly cover carpenter. I would also recommend that you do a deeper dive into into carpenter with the technical lead Ellis turn from carpenter in the future. I can always connect the two of you together. But carpenter drew Out of I don't know how familiar you are with the upstream Kubernetes Cluster Autoscaler. But it is a multi cloud provider solution that enables the Kubernetes worker nodes, the pool of Kubernetes worker nodes to be expanded or shrunk. Depending on for expanding its depending on the number of pending pods that are ready to be scheduled, right? That are have gone into the scheduler and there is no room on the existing pool of worker nodes for those workloads to go on. And so the Cluster Autoscaler needs to find a set of nodes, a node that will be able to be expanded so that those unscheduled workloads can can land on there. The upstream Cluster Autoscaler has drivers for GCP and Azure and AWS and Equinix, and lots of lots of providers. But the upstream Cluster Autoscaler uses this, this concept of a node group, right. And the note group is a uniform set of worker nodes that have the same instance size, right, like the number of V CPUs, memory, and so on. And so all worker nodes within a note group have the exact same size. When the implementation for the AWS provider inside Cluster Autoscaler was made, that we made the choice to use the EC to auto scaling groups API. And that API, the Auto Scaling group API is sort of in congruent with the idea that a node group has to have uniform, homogenous instance types, right. Because with an easy to alter Scaling group, you could specify a number of instance types that can have vastly different number of V CPUs and memory deaths associated that has caused friction inside the AWS provider and Cluster Autoscaler. What carpenter originally was designed to do was to remove that friction by having the carpenter provider for provisioner, called Easy to fleet API, instead of the Auto Scaling group API. The fleet API is a more fine grained approach that lets you essentially give the range of instance types and V CPUs and memory that you need. And it will just pick you at the instance type that matches those criteria, but not put it into a group. Right. And so this constraint of having to tightly group and couple instances together that Cluster Autoscaler paths upstream carpenter got rid of that constraint. And it's able to do much finer grained placement decisions than the upstream Cluster Autoscaler. It also there's a couple of major differences between Cluster Autoscaler and Carpenter, and one of which is Cluster Autoscaler actually embeds the Kubernetes core scheduling system inside it. And it does this in order to run experiments, right to determine if, for instance, cluster auto scaling will say, if I scale up this particular node group, with two more nodes, will I be able to schedule or place these pending pods on that set of notes? Carpenter does not do that. It doesn't actually emulate scheduling, it actually goes ahead and schedules things. And so it binds a pending pod to a worker now that it is spun up using the fleet API in the case of AWS directly. And so it sort of bypasses, the scheduler and a lot twigs, it also has some abilities to consolidate kind of like a deep like disk. defrag remember, like, when you ran your disk defrag. And you see, like, going consolidating things, Carpenter has a very similar functionality to repack, pots on worker nodes, because you know, over time, so holes start to appear in worker notes as pots are terminated. And, and so like, you know, it gets a little fragmented. And so Carpenter has this ability to sort of defrag that, that also is a sort of like a full lifecycle node lifecycle management system. So it can handle like upgrades of the Amazon machine images, and draining and coordinating process and things like that. So in a way, it's Cluster Autoscaler and redefined with note termination handler, and the D scheduler from upstream Kubernetes kind of all rolled into one, but it does have a provider model, very similar to Cluster Autoscaler upstream so that it's not just AWS specific right now, the only provider is for EC two, but we hope to have Have other providers for Azure and Google etc.
Joab Jackson 15:03
Now, so I'm just kind of curious, I don't know if you're in a position to answer this. But why wouldn't AWS keep that as its secret sauce for its customers, like open source for the competitors to use?
Jay Pipes 15:15
Well, we don't really see this as our special sauce, right? I mean, at the end of the day, it's just calling an upstream fleet API, right, which is public and open. Anyone can can call call fleet API, where AWS sees its value is providing managed offerings for things. So you know, we just like Eks is a is a managed service for Kubernetes control points. Right, Carpenter, you still have to install it yourself, right, and run it in your cluster and operate it. Right. What AWS sees as our value add is in managing and running those types of things for you. So I think it's safe to say that, you know, you will see in the future AWS offering carpenter as soon as like a managed offering inside eks. Right. Nice. Nice.
Joab Jackson 16:08
Interesting. All right, because they could also talk about CDK. S, this is the AWS CDK. It's the cloud development kit for you.
Jay Pipes 16:20
Right? So yeah, so CDK is this AWS specific CloudFormation synthesis, sort of toolkit, right. And CD case, is very similar to CDK. And it's feel, right, you can develop using TypeScript and go and your normal regular favorite programming language, but instead of outputting, CloudFormation templates, what CD Cates does is outputs Kubernetes manifests. So you can like programmatically deal with pods and services and pin graphs and all of these kinds of objects, using TypeScript. Right. And it has like nice IDE integration, and you know, like syntax highlighting, auto completion, all that kind of stuff. But at the end of the day, he didn't CDK synthesize those things into the Yamo that Kubernetes expects, so what just like CDK, it's the same thing. But what gets output is the CloudFormation template that you will, you know, execute as a step. But for CDK, it's, it's the same idea, but you're working with these Kubernetes objects, and it outputs the Yamo that you will then come apply.
Joab Jackson 17:31
Nice. So it's very handy. It is
Jay Pipes 17:35
personally, a bit of a Yeah, I like there's a ton of books out there. There's like hate Gamble's and all that. And I'm not one of them. So like, I could deal with the devil all day long. But there's there's plenty of people who just prefer to use their normal programming language and not have to deal with like indentation and syntax issues and gamble. So syndicates is a great tool. Nice, nice.
Joab Jackson 18:01
So melt. Obviously Kubernetes is very important, AWS, and we cover the Cloud Native Computing Foundation. Quite a bit. You know, we were checking out what was happening at cube con. What other cn CF projects does AWS take a close interest in, in the CMC F realm?
Jay Pipes 18:20
Yeah, so we have core maintainers on Container D fluid OTEL, obviously Kubernetes, we have maintainers, or at least where we're trying to get containers for at CD. Because if you think about producing a managed Kubernetes service at scale, is basically the proper care and feeding of Sed. Like we have a CD footprint, right? Because each of the Kubernetes clusters that are vetted by Eks, that comes along with a multi node, sed cluster. And so, you know, for the many, many, many 1000s of Kubernetes clusters that we manage, we we are managing a sed cluster for a specific customer. And so we have a lot of operational experience, running sed scaling, sed snapshotting, restoring and all that kind of stuff with sed at scale. And so we're we have two engineers that are working to be full time maintainers on on Etsy D, we have full time maintainers on cortex, which is also a CNC F I believe incubating project. So yep, yeah. So yeah, we're kind of all over the place. We have we have maintainers and lots of these different projects. Some might say, we are we are spread quite thinly, I guess across a lot of these projects, but some of the Like for instance, container D and we have three active maintainers and we're the number one maintainer in this last year and container D so yeah, we we contribute quite heavily to a lot of the CN CF projects
Joab Jackson 19:54
nice nice so you yourself lead the AWS controllers for Kubernetes project could Talk a bit about
Jay Pipes 20:00
that. Sure. I do, I will point out that is not the CN CF project. Okay. So basically is it's a set of Kubernetes custom controllers that Kubernetes users can use to interact with an AWS service. So for instance, if I want to create an RDS database instance, I can use the RDS controller in a ck, and I can just call coop cuddle apply pass in some Yamo. And the controller, the RDS controller for AC K will talk to RDS set up the database instance and allow you to manipulate the RDS DB instance just using Kubernetes. Gamble. So it's like a bridge between the Kubernetes API world and the AWS service API world. We've got like 23 controllers now, I think 12 or 13, or ngaa, the rest are in developer preview. So yeah, we're adding new services all the time, we work with individual AWS service teams, in writing their controller, and most of the controller is actually code generated, right. So it's roughly 95 96% of, of all of the Go code that comprises these individual SDK controllers. This January, we've read the API model definition, and we run it through a bunch of cogeneration. And output, the actual controller implementation. So it's, it's pretty heavy on the cogeneration side, but we do work with AWS service teams, because they know their API best, right. And so we work with the AWS service teams, to help them develop their own controller and take over maintenance.
Joab Jackson 21:33
So for the end user, the AWS developer or admin, this would allow them to, I guess, provision AWS services directly through Kubernetes, or what I described,
Jay Pipes 21:46
that's correct, like, so. Yes, if you, for instance, don't want to run Postgres in your own Kubernetes cluster, and is that just want to rely on the auto managed RDS service, right, you can define a Yamo manifest right in the SDK custom resource for RDS DB instance, specify the engine bone, the engine title and tech stuff, and the size and you know, whether that's provision serverless, all that kind of stuff, and then just call coop cuddle apply, passing in that definition. And behind the scenes, the controller is going to be making a bunch of calls to RDS to create the database itself, modify its attributes and set parameter groups. So it just kind of allows Kubernetes users and application developers to stay in their comfy little Kubernetes world and not have to go to the AWS console, not have to use the AWS CLI or anything, they could just stay in the Kubernetes world. And everything just kind of works for at least that's like that's the goal. So
Joab Jackson 22:46
nice. Nice. All the paperwork is taken care of. So now let's talk a bit more about cube con. So you're you're on the CN CF governing board, right?
Jay Pipes 22:56
I am indeed Yes, I am the AWS representative as the Platinum Member representative from from AWS, I assumed Bob wises seat when Bob left AWS in February,
Joab Jackson 23:09
both those so I thought we should mention her. But what was your takeaway from cube? Con? What are the communities? What were the technologies to watch your what struck you is interesting at this year's show?
Jay Pipes 23:21
Well, Joe, I'm glad you asked that, because I had forgotten to bring this up earlier when we're talking about open source contribution, everything. So the biggest topic at the strategy session for the CN CF governing board this year at coop con in Detroit a couple weeks ago, was the upstream Kubernetes CI system and image registry costs. So right now, the Kubernetes Container Registry. So it's been housed at Kate stop gcr.io, which the Google Container Registry and all of the CI infrastructure that we will use in the upstream Kubernetes development runs on Google on GCP. We have been much long effort to reduce the costs borne by Google solely for the CI system and the Container Registry and spread that amongst the other contributing organizations to Kubernetes. Just to give you an idea of some like the dollar amounts here. So like container registry that stores all of the Kubernetes container images, right the cube API server coop proxy core DNS, all those kinds of things. It's about 1.5 terabytes of storage data in GCR. On a monthly case, we are consuming 90 to $100,000 in egress bandwidth costs out of Google data centers and into Amazon data centers, because you've got a bunch of EC two instances that are running cubelet, the worker node agent that are pulling container images out of Google's data centers and into AWS. So that 100,000 times dollars a month and egress costs was eating into the total, the credits that Google has provided CNCF, about $3 million in credits. And so we've been working on a solution that mirrors the container images, and we'll dive into the blobs, the container image layers, mirrors those into s3 buckets in the 12, most popular AWS regions. And by doing that, we also set up this, this proxy called registry dot Kate's dot IO that can tell when a caller for a particular container image just coming from an easy to instance, and if so, redirect the image layers to that s3 mirror that's running inside that AWS region. By doing that, we completely eliminate the egress costs out of Google data centers into AWS data centers. So once rather than Yeah, well, no, because there is no costs for an easy to instance, to read from an s3 bucket in its local region. And so so we remove all of that cost. Anyway, this is one area that we're we're working closely with the upstream suitcase in front in Kubernetes, to reduce these costs, and have AWS bear more of those costs over time. Also, Barry copes, who's the VP of Kubernetes, at AWS announced AWS Cloud Credits up to $3 million a year and full time engineering resources to dedicate to the core common infrastructure needs and testing infrastructure needs of Kubernetes. This was a huge deal. And it was announced on Thursday that in a tapers keynote, and we discussed it later on Thursday in the strategy session with cn CF governing board. But I'm really hopeful that this is sort of like the catalyst, the impetus for renewed collaboration between the major cloud providers to share more of the costs for our Upstream open source project infrastructure than just Google. Right,
Joab Jackson 27:03
there's so the work would not just be for AWS, but for any cloud provider or even large enterprise, if they if they want to do something similar, I guess,
Jay Pipes 27:14
exactly, Joe. I mean, right now, around 70% of all these image polls are coming from AWS, because that's, I mean, I know we got such a large percentage of the Kubernetes workloads running on ECE two. And so for this first, sort of go around, we're aiming to cut the most amount of costs by setting up these s3 mirrors. But, you know, we're, we're going to do the same in the future for Azure. And just to spread the cost more around and say, with running the CI infrastructure, something called trial upstream currently runs only on GCP, where we have efforts underway to bring that out, only GCP and share the burden amongst the three major cloud providers.
Joab Jackson 27:54
Is there a worry that Google is handling too much of the infrastructure support?
Jay Pipes 28:00
I mean, I'm personally worried about burnout of Google engineers that that currently, because everything's inside GCP, at least from a CI stuff, it is a burden on GCP engineers, and I personally worry about them burning out, because they're single points of contact for some of these systems. So there isn't any danger of like Google walking away from Kubernetes, or anything like that. I'm more concerned about just the personnel and the individual contributors that are sort of heroically keeping all of this stuff going. I would like it to not be heroic thing, right, and spread the burden across engineering resources that are borne by all the major cloud providers. All right, terrific.
Joab Jackson 28:44
Terrific. Well, we want to in this interview, we wanted to talk about AWS and it's open source contributions to the cloud native community. And I got a good overview, I was unaware that there was so much more going on. Are there any other aspects you think might be worth mentioning to people who are interested in this topic?
Jay Pipes 29:02
Well, I think you know, there are a ton of AWS engineers and solutions architects and product managers that hang out on the Kubernetes community slack and the CN CF community slack comm find us right, there's, there's lots of channels, there's provider AWS chat on carpenter channel, the there's the AWS controllers for Kubernetes channel on the Kubernetes slack comm find us and tell us what you'd like us to work on. Or if you have a particular issue that you've found in one of these upstream projects that you think our engineers can help move the needle on. Come, come find us. Come talk to us. All right, fantastic.
Joab Jackson 29:37
Jay, thank you so much for taking time to talk with us. And thank you listeners and viewers for tuning into the new stack makers podcast. And we'll see you all shortly.
Colleen Coll 29:51
Amazon Web Services is the world's most comprehensive and broadly adopted cloud platform, offering over 175 fully featured services from datacenters. Globally, millions of customers trust AWS to power their infrastructure become more agile and lower costs.
Alex Williams 30:11
Thanks for listening. If you liked the show, please rate and review us on Apple podcast Spotify, or wherever you get your podcasts. That's one of the best ways you can help us grow this community and we really appreciate your feedback. You can find the full video version of this episode on YouTube. Search for the new stack and don't forget to subscribe so you never miss any new videos. Thanks for joining us and see you soon.
Transcribed by https://otter.ai