Kubernetes is much more than container scheduling. Custom resources, identity and a powerful RBAC system allow you to use it as a general configuration/operational data store in your own code, from implementing operators acting upon kubernetes and the outside world to even moving most of high-level configuration glue to be natively based on Kubernetes.
For example, with cert-manager running on Kubernetes you can request a TLS certificate by creating a Certificate resource (like you would any other Kubernetes resource). This is the same regardless of whether you want a self-signed certificate, an ACME-issued certificate (and whether that gets performed via HTTP01 or DNS01 or something else). Oh, and this fully ties into Kubernetes' RBAC system.
In Nomad the closest thing is annotating jobs with traefik-specific tags (and allowing Traefik to do cluster-wide discovery of all tags), but that only works for serving certificates that are managed by traefik, not if your application wants eg. to terminate the TLS connection itself, or if it wants some other PKI hierarchy (eg. a self-signed CA which then issues some other certificates for mutual TLS auth between application services).
Kubernetes also has better support for organization-wide multi-tenant clusters than Nomad seems to have (eg. nomad's policy engine, audit logging and resource quota system are gated behind their “enterprise” offering).
> Kubernetes also has better support for organization-wide multi-tenant clusters than Nomad seems to have
That one's a little weird, I suppose you're right, but all the clients our Kubernetes team works with all want separate clusters or testing, staging and preproduction. They certainly don't want a multi-tenant cluster and share resources with other clients.
> but all the clients our Kubernetes team works with all want separate clusters or testing, staging and preproduction.
And I think that's one of the biggest issues with how people use Kubernetes these days (another candidate being insisting on drive-by deploying a cluster from scratch instead of deferring to cloud providers or a dedicated platform team that can plan for long-term maintenance).
Kubernetes thrives in multi-tenant environments: you get huge resource savings and vastly simplified operations. Everyone in your organization gets access to all clusters, and they can just as easily deploy experimental best effort jobs or their development environment as they can deploy and/or inspect production jobs. Well set up quotas and priority classes mean that production jobs never run out of resources, while less important stuff (batch jobs, CI, someone's pet experiment) can continue to run on a best effort basis, just keeps getting preempted when production wants more resources.
You can even continue to have hardware separation between highly sensitive and fully untrusted jobs by using taints and tolerations, if you feel that's necessary. You still get one control plane instead of five different ones.
> And I think that's one of the biggest issues with how people use Kubernetes these days (another candidate being insisting on drive-by deploying a cluster from scratch instead of deferring to cloud providers or a dedicated platform team that can plan for long-term maintenance).
I don't really understand how you can say this and then...
> Kubernetes thrives in multi-tenant environments: you get huge resource savings and vastly simplified operations. Everyone in your organization gets access to all clusters, and they can just as easily deploy experimental best effort jobs or their development environment as they can deploy and/or inspect production jobs. Well set up quotas and priority classes mean that production jobs never run out of resources, while less important stuff (batch jobs, CI, someone's pet experiment) can continue to run on a best effort basis, just keeps getting preempted when production wants more resources.
... advocate for this. All what you are describing, which is basically what every hardcore k8s user/evangelist will tell you to do, it's reimplementing many, if not all, the features a Cloud provider is already giving you in their own resources. But you are taking the ownership and responsibility for this on your local platform/infra team. What if you screw something with CoreDNS? what if you break some RBAC roles used cluster-wide, while trying a change in the beta environment? I'm pretty sure there are (or will be) specific k8s tools to manage this but still, you are adding complexity and basically running another cloud provider inside a cloud provider for the sake of binpacking. For certain sizes of companies it might be worth the effort, but it is for sure not a silver bullet and probably applies to much less companies that many evangelists try to sell.
> ... advocate for this. All what you are describing, which is basically what every hardcore k8s user/evangelist will tell you to do, it's reimplementing many, if not all, the features a Cloud provider is already giving you in their own resources.
A well-designed KaaS offering from a cloud provider will do that by itself. GKE exposes GCP load balancers as an Ingress controller, IAM identities as Kubernetes RBAC identities, persistent disks as PVs, ... You just get them under a single declarative API.
> But you are taking the ownership and responsibility for this on your local platform/infra team.
With a platform team you're concentrating already existing responsibility into a team that can specialize in operational excellence - vs. that same responsibility being spread out across product teams that have to individually manage their own cloud resources, reinventing the wheel by writing the same terraform/{ansible,puppet,chef,...} boilerplate poorly. My experience is that these per-team bespoke AWS deployments are much more brittle than whatever a dedicated team can provide if given the responsibility and means to do things well.
> What if you screw something with CoreDNS? what if you break some RBAC roles used cluster-wide, while trying a change in the beta environment?
An outage is an outage, you roll back to stop the bleeding, investigate what happened and try to prevent whatever caused it from happening in the future. Neither of these examples are unsolvable in a multi-tenant environment, nor especially more likely to happen than similar screwups when using cloud provider resources.
> A well-designed KaaS offering from a cloud provider will do that by itself. GKE exposes GCP load balancers as an Ingress controller, IAM identities as Kubernetes RBAC identities, persistent disks as PVs, ... You just get them under a single declarative API.
My experience with EKS on AWS tells me that it's not that simple, there are still many things to be glued together. I understand AWS historical position on K8s, and they probably want to keep the k8s experience on AWS good but not awesome but I'm pretty sure that there are even in GCP still serious gaps between "native" GCP features and k8s ones, where you end up reimplementing them on both sides. But I'm no GCP expert so I might be totally wrong.
> With a platform team you're concentrating already existing responsibility into a team that can specialize in operational excellence - vs. that same responsibility being spread out across product teams that have to individually manage their own cloud resources, reinventing the wheel by writing the same terraform/{ansible,puppet,chef,...} boilerplate poorly. My experience is that these per-team bespoke AWS deployments are much more brittle than whatever a dedicated team can provide if given the responsibility and means to do things well.
I'm totally fine with this approach, and we are actually trying to implement it at $DAYJOB but I don't really get why you see thew AWS API as a different monster from the K8s API. With a complex enough system you will need many lines of YAML/charts/Terraform/whatever on the k8s just like CF/Terraform/Pulumi/whatever on AWS. And you can totally have a team that takes care of the quirks and details of AWS while exposing a usable and unified interface for services deployements to the the rest of the engineering organization. I understand if we were talking about bare metal vs Kubernetes (even on-prem), k8s would win hands-down. But in the cloud-native world, I don't really see that day vs night change. Everything has its tradeoffs and its quirks and bugs and corner cases.
With things like custom operators, especially crossplane (but also anything custom you cook up fast) or even custom operator wrapping AWS or GCP templates it's easy for me to offer curated verified solutions across all teams, instead of every one hacking off their own AWS/GCP/WTFcloud scripts to handle things. Even better than directly using cloud provider integration with ingress/service controllers, because I can provide specific limited variants of those APIs. And even without that, I can just use hooks system to blunt corners for the teams.
> You can even continue to have hardware separation between highly sensitive and fully untrusted jobs by using taints and tolerations, if you feel that's necessary. You still get one control plane instead of five different ones.
How much have you had that setup audited? It seems like a lot people aren’t comfortable saying that the internal boundaries are strong enough, which leads to the proliferation of separate clusters.
For example, with cert-manager running on Kubernetes you can request a TLS certificate by creating a Certificate resource (like you would any other Kubernetes resource). This is the same regardless of whether you want a self-signed certificate, an ACME-issued certificate (and whether that gets performed via HTTP01 or DNS01 or something else). Oh, and this fully ties into Kubernetes' RBAC system.
In Nomad the closest thing is annotating jobs with traefik-specific tags (and allowing Traefik to do cluster-wide discovery of all tags), but that only works for serving certificates that are managed by traefik, not if your application wants eg. to terminate the TLS connection itself, or if it wants some other PKI hierarchy (eg. a self-signed CA which then issues some other certificates for mutual TLS auth between application services).
Kubernetes also has better support for organization-wide multi-tenant clusters than Nomad seems to have (eg. nomad's policy engine, audit logging and resource quota system are gated behind their “enterprise” offering).