Terraform:如何使用命名空间在 Google Cloud (GKE) 上创建 Kubernetes 集群? [英] Terraform: How to create a Kubernetes cluster on Google Cloud (GKE) with namespaces?

问题描述

我正在寻找一个可以执行以下操作的示例:

I'm after an example that would do the following:

1. 通过 Terraform 的 google_container_cluster<在 GKE 上创建 Kubernetes 集群/代码>
2. ... 并继续在其中创建命名空间，我想通过 kubernetes_namespace

我不确定的是如何连接新创建的集群和命名空间定义.例如，当添加 google_container_node_pool 时，我可以执行类似 cluster = "${google_container_cluster.hosting.name}" 的操作，但对于 我没有看到类似的操作kubernetes_namespace.

The thing I'm not sure about is how to connect the newly created cluster and the namespace definition. For example, when adding google_container_node_pool, I can do something like cluster = "${google_container_cluster.hosting.name}" but I don't see anything similar for kubernetes_namespace.

推荐答案

理论上可以在 K8S(或任何其他)提供程序中引用 GCP 提供程序的资源，就像在其中引用资源或数据源一样单个提供者的上下文.

In theory it is possible to reference resources from the GCP provider in K8S (or any other) provider in the same way you'd reference resources or data sources within the context of a single provider.

provider "google" {
  region = "us-west1"
}

data "google_compute_zones" "available" {}

resource "google_container_cluster" "primary" {
  name = "the-only-marcellus-wallace"
  zone = "${data.google_compute_zones.available.names[0]}"
  initial_node_count = 3

  additional_zones = [
    "${data.google_compute_zones.available.names[1]}"
  ]

  master_auth {
    username = "mr.yoda"
    password = "adoy.rm"
  }

  node_config {
    oauth_scopes = [
      "https://www.googleapis.com/auth/compute",
      "https://www.googleapis.com/auth/devstorage.read_only",
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring"
    ]
  }
}

provider "kubernetes" {
  host = "https://${google_container_cluster.primary.endpoint}"
  username = "${google_container_cluster.primary.master_auth.0.username}"
  password = "${google_container_cluster.primary.master_auth.0.password}"
  client_certificate = "${base64decode(google_container_cluster.primary.master_auth.0.client_certificate)}"
  client_key = "${base64decode(google_container_cluster.primary.master_auth.0.client_key)}"
  cluster_ca_certificate = "${base64decode(google_container_cluster.primary.master_auth.0.cluster_ca_certificate)}"
}

resource "kubernetes_namespace" "n" {
  metadata {
    name = "blablah"
  }
}

但实际上它可能无法按预期工作，因为一个已知的核心错误会破坏跨提供商的依赖关系，请参阅 https://github.com/hashicorp/terraform/issues/12393 和 https分别为://github.com/hashicorp/terraform/issues/4149.

However in practice it may not work as expected due to a known core bug breaking cross-provider dependencies, see https://github.com/hashicorp/terraform/issues/12393 and https://github.com/hashicorp/terraform/issues/4149 respectively.

替代解决方案是:

1. 使用 2 阶段应用并定位 GKE 集群首先，然后是其他任何依赖它的东西，即 terraform apply -target=google_container_cluster.primary 然后 terraform apply
2. 将 GKE 集群配置从 K8S 配置中分离出来，为它们提供完全隔离的工作流，并通过 连接它们远程状态.

1. Use 2-staged apply and target the GKE cluster first, then anything else that depends on it, i.e. terraform apply -target=google_container_cluster.primary and then terraform apply
2. Separate out GKE cluster config from K8S configs, give them completely isolated workflow and connect those via remote state.

/terraform-gke/main.tf

terraform {
  backend "gcs" {
    bucket  = "tf-state-prod"
    prefix  = "terraform/state"
  }
}

provider "google" {
  region = "us-west1"
}

data "google_compute_zones" "available" {}

resource "google_container_cluster" "primary" {
  name = "the-only-marcellus-wallace"
  zone = "${data.google_compute_zones.available.names[0]}"
  initial_node_count = 3

  additional_zones = [
    "${data.google_compute_zones.available.names[1]}"
  ]

  master_auth {
    username = "mr.yoda"
    password = "adoy.rm"
  }

  node_config {
    oauth_scopes = [
      "https://www.googleapis.com/auth/compute",
      "https://www.googleapis.com/auth/devstorage.read_only",
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring"
    ]
  }
}

output "gke_host" {
  value = "https://${google_container_cluster.primary.endpoint}"
}

output "gke_username" {
  value = "${google_container_cluster.primary.master_auth.0.username}"
}

output "gke_password" {
  value = "${google_container_cluster.primary.master_auth.0.password}"
}

output "gke_client_certificate" {
  value = "${base64decode(google_container_cluster.primary.master_auth.0.client_certificate)}"
}

output "gke_client_key" {
  value = "${base64decode(google_container_cluster.primary.master_auth.0.client_key)}"
}

output "gke_cluster_ca_certificate" {
  value = "${base64decode(google_container_cluster.primary.master_auth.0.cluster_ca_certificate)}"
}

在这里，我们通过 output 公开所有必要的配置，并使用后端将状态以及这些输出存储在远程位置，GCS 在这种情况下.这使我们能够在下面的配置中引用它.

Here we're exposing all the necessary configuration via outputs and use backend to store the state, along with these outputs in a remote location, GCS in this case. This enables us to reference it in the config below.

/terraform-k8s/main.tf

data "terraform_remote_state" "foo" {
  backend = "gcs"
  config {
    bucket  = "tf-state-prod"
    prefix  = "terraform/state"
  }
}

provider "kubernetes" {
  host = "https://${data.terraform_remote_state.foo.gke_host}"
  username = "${data.terraform_remote_state.foo.gke_username}"
  password = "${data.terraform_remote_state.foo.gke_password}"
  client_certificate = "${base64decode(data.terraform_remote_state.foo.gke_client_certificate)}"
  client_key = "${base64decode(data.terraform_remote_state.foo.gke_client_key)}"
  cluster_ca_certificate = "${base64decode(data.terraform_remote_state.foo.gke_cluster_ca_certificate)}"
}

resource "kubernetes_namespace" "n" {
  metadata {
    name = "blablah"
  }
}

这里可能不明显的是，必须在创建/更新任何 K8S 资源之前创建/更新集群(如果此类更新依赖于集群的更新).

What may or may not be obvious here is that cluster has to be created/updated before creating/updating any K8S resources (if such update relies on updates of the cluster).

无论哪种方式，通常都建议采用第二种方法(即使/如果错误不是一个因素并且跨提供商参考有效)，因为它可以减少爆炸半径并定义更清晰的责任.此类部署通常由 1 个人/团队负责管理集群，而由不同的个人/团队负责管理 K8S 资源，这是 (IMO) 常见的.

Taking the 2nd approach is generally advisable either way (even when/if the bug was not a factor and cross-provider references worked) as it reduces the blast radius and defines much clearer responsibility. It's (IMO) common for such deployment to have 1 person/team responsible for managing the cluster and a different one for managing K8S resources.

当然可能会有重叠 - 例如想要部署日志记录的运维人员在新的 GKE 集群之上监控基础设施，因此跨提供者依赖项旨在满足此类用例.因此，我建议订阅上述 GH 问题.

There may certainly be overlaps though - e.g. ops wanting to deploy logging & monitoring infrastructure on top of a fresh GKE cluster, so cross provider dependencies aim to satisfy such use cases. For that reason I'd recommend subscribing to the GH issues mentioned above.

这篇关于Terraform:如何使用命名空间在 Google Cloud (GKE) 上创建 Kubernetes 集群?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！