Ingress controller in k8s cluster on GCP by Terraform. Part 2.

Incoming traffic (ingress) management in GKE by Terraform and CI/CD pipeline.

In the first Part 1 we created the source code structure and first GKE cluster. Let’s assume that we want to create a simple web-site in Internet. The primary requirement is not to serve only HTTP traffic, but also to request and issue Let’s Encrypt TLS certificates, create endpoints and serve HTTPS traffic.

There are two options to organize the ingress in k8s:

1. Use Ingress-Controller
2. Use new Gateway API

Believe me or not, I spent a lot of time trying to make things working by using modern Gateway API in GCP/GKE with their own controllers: gke-l7-global-external-managed and gke-l7-regional-external-managed that requires a lot of steps:

• Enable Gateway API in GKE cluster
• Install Gateway API CRDs
• Define the GatewayClass, Gateway, HTTRouter resources
• Use ManagedCertificate
• Use CertMap
• Use LoadBalancer for gateway controller
• Use regional network for regional controller
• Use Cert-Manager since ManagedCertificates are not working
• Too sensitive and limited gateway api controller

But I was not able to achieve automatic Let’s Encrypt / GCP certificates generation for the GCP Endpoints. Therefore my primary option became to use the legacy open source ingress controller, it just works.

Another benefit of using the legacy ingress controller —to be independent from the particular k8s cluster implementation.

From the existing Ingress controllers I selected the most popular one: ingress-nginx. It works very well with cert-manager.

Prerequisites

• Project Name — devops-example-project
• Cluster Name — devops-cluster
• Already created source code project in Part 1.
• Already created GKE cluster in Part 1.

Terraform

Let’s create ingress folder in our source code project and terraform files.

cd terraform/ingresscat > .terraform-version1.12.1

The fixed terraform version helps to keep things organized and reproducible.

cat > versions.tfterraform {  required_providers {    google = {      source  = "hashicorp/google"      version = "~> 6.44"    }    helm = {      source  = "hashicorp/helm"      version = "~> 2.0"    }  }}

The most important here is the version of helm plugin.

Let’s start with terraform backend

terraform {  backend "gcs" {    bucket  = "devops-terraform-state" # You must create this bucket first    prefix  = "devops-cluster/ingress"  }}

The bucket we already created in Part 1. The prefix must be unique and not intercept with any other one, since terraform would keep here state for ingress resources only.

Let’s add variables that would be used in this config

cat > variables.tfvariable "project_id" {  type    = string  default = "devops-example-project"}variable "region" {  type    = string  default = "us-central1"}variable "region_with_zone" {  type    = string  default = "us-central1-a"}variable "cluster_name" {  type    = string  default = "devops-cluster"}variable "email" {  type    = string  default = "your_email@your_host.com"}

The email variable is needed for the Let’s Encrypt account. Make sure that it is your working email or group email, you will get notifications from Let’s encrypt about your certificates.

Let’s define the main.tf file that has a reference to the GKE cluster that we’v created in Part 1. For that we are using the data resource that has only reference to the existing cluster. Remember, I specially separated cluster creation to decrease the risks of operations.

cat > main.tf# Configure the Google Cloud providerprovider "google" {  project = var.project_id  region  = var.region}# Data source for Google client configdata "google_client_config" "default" {}data "google_container_cluster" "gke" {  name     = var.cluster_name  location = var.region_with_zone}# Configure the Kubernetes providerprovider "kubernetes" {  host                   = "https://${data.google_container_cluster.gke.endpoint}"  cluster_ca_certificate = base64decode(data.google_container_cluster.gke.master_auth[0].cluster_ca_certificate)  token                  = data.google_client_config.default.access_token}# Configure the Helm providerprovider "helm" {  kubernetes {    host                   = "https://${data.google_container_cluster.gke.endpoint}"    cluster_ca_certificate = base64decode(data.google_container_cluster.gke.master_auth[0].cluster_ca_certificate)    token                  = data.google_client_config.default.access_token  }}

Helm provider would be needed to install ingress controller and cert-manager.

For the external web-site we will need the static IP address.

cat > external-ip.tf# Reserve a global static IP addressresource "google_compute_address" "external_ip" {  name   = "devops-cluster-dev-static-ip"  region = var.region}

This is a regional IP address, since we have only one GKE cluster in one region/zone.

Let’s define the ingress controller:

cat > ingress.tfresource "kubernetes_namespace" "ingress_nginx" {  metadata {    name = "ingress-nginx"  }  lifecycle {    ignore_changes = [metadata]  }}# Install NGINX Ingress Controller using Helmresource "helm_release" "ingress_nginx" {  name             = "ingress-nginx"  repository       = "https://kubernetes.github.io/ingress-nginx"  chart            = "ingress-nginx"  version          = "4.12.4"  namespace        = kubernetes_namespace.ingress_nginx.metadata[0].name  create_namespace = false    # This mimics the --upgrade --install behavior  force_update    = false  recreate_pods   = false  cleanup_on_fail = true  # Increase timeout for GKE deployment  timeout = 900  # 15 minutes  wait    = true  # Configure static external IP (regional)  set {    name  = "controller.service.loadBalancerIP"    value = google_compute_address.external_ip.address  }  # Ensure service type is LoadBalancer  set {    name  = "controller.service.type"    value = "LoadBalancer"  }  depends_on = [    kubernetes_namespace.ingress_nginx,    google_compute_address.external_ip  ]}

We also need cert-manager for this ingress to issue certificates

cat > cert-manager.tf# Create cert-manager namespaceresource "kubernetes_namespace" "cert_manager" {  metadata {    name = "cert-manager"  }  lifecycle {    ignore_changes = [metadata]  }}# Install cert-manager using Helmresource "helm_release" "cert_manager" {  name       = "cert-manager"  repository = "https://charts.jetstack.io"  chart      = "cert-manager"  version    = "v1.17.4"  namespace  = kubernetes_namespace.cert_manager.metadata[0].name  set {    name  = "installCRDs"    value = "false"  }  depends_on = [kubernetes_namespace.cert_manager]}resource "kubernetes_manifest" "letsencrypt_cluster_issuer" {  manifest = {    apiVersion = "cert-manager.io/v1"    kind       = "ClusterIssuer"    metadata = {      name = "letsencrypt-prod"    }    spec = {      acme = {        email = var.email        server = "https://acme-v02.api.letsencrypt.org/directory"        privateKeySecretRef = {          name = "letsencrypt-prod"        }        solvers = [          {            http01 = {              ingress = {                class = "nginx"              }            }          }        ]      }    }  }  depends_on = [helm_release.cert_manager]}

For the cert-manager I specially selected option installCRDs=false since I would like to install them manually. By doing this we are removing class-level CRDs from managing by Terraform. Otherwise we would have a lot of installation, update, upgrade issues with endless fails in terraform apply.

Let’s install CRDs manually

# login to GKE clustergcloud container clusters get-credentials devops-cluster \  --zone us-central1-a \  --project $PROJECTcurl -L -o cert-manager-crds.yaml https://github.com/cert-manager/cert-manager/releases/download/v1.17.4/cert-manager.crds.yamlkubectl apply -f cert-manager.crds.yaml

And finally, let’s define the outputs for the terraform

cat > outputs.tf# Output the external IP addressoutput "nginx_ingress_ip" {  description = "The external IP address of the NGINX Ingress Controller"  value       = google_compute_address.external_ip.address}# Output the ingress class nameoutput "ingress_class_name" {  description = "The name of the ingress class"  value       = "nginx"}

After executing the terraform commands we should get External IP address under the nginx_ingress_ip.

But before running the terraform scripts from the console, we need to authenticate by GCP

gcloud auth logingcloud config set project devops-example-project

and now we are ready to run scripts in the repo

cd terraform/ingressterraform initterraform planteraform apply

After executing our scripts at the first time we should create in k8s

• namespace ingress-nginx with installed ingress controller
• namespace cert-manager with installed certificate manager
• static ip address external_ip

We can use External IP in DNS manager for any domain name or can create GCP Endpoint and assign External IP address to it. Both options works well.

You can login to your GKE cluster and verify the installation

# login to GKE clustergcloud container clusters get-credentials devops-cluster \  --zone us-central1-a \  --project $PROJECTkubectl get namespaces

You should have two namespaces created: ingress-nginx, cert-manager.

Wait 1–2 minutes…

Your external_ip address should match the ingress controller External IP address:

kubectl get ingress -n ingress-nginx

CI/CD pipeline

As of now we are ready to automate updates by using GitHub Actions.

In our source code project create the file .github/workflows/deploy-ingress.yaml that will automatically apply terraform changes in the GKE cluster from our ingress folder in the main branch.

name: Build and Push to GCPon:  push:    branches:      - mainenv:  PROJECT_ID: devops-example-project  REGION_WITH_ZONE: us-central1-a  CLUSTER_NAME: devops-clusterjobs:  terraform-deploy:    runs-on: ubuntu-latest    defaults:      run:        working-directory: terraform/gateway    steps:      - name: Checkout source code        uses: actions/checkout@v3      - name: Set up Terraform        uses: hashicorp/setup-terraform@v2        with:          terraform_version: 1.12.1      - name: Authenticate to Google Cloud        uses: google-github-actions/auth@v2        with:          credentials_json: '${{ secrets.GCP_DEPLOYMENTS_SA_KEY }}'      - name: Get GKE Credentials        run: |          gcloud container clusters get-credentials ${{ env.CLUSTER_NAME }} \            --region ${{ env.REGION_WITH_ZONE }} \            --project ${{ env.PROJECT_ID }}      - name: Terraform Init        run: terraform init      - name: Terraform Plan        run: terraform plan      - name: Terraform Apply        run: terraform apply -auto-approve

Use your own PROJECT_ID, REGION_WITH_ZONE, CLUSTER_NAME variables.

This workflow action is using the GitHub secret GCP_DEPLOYMENTS_SA_KEY that contains the GCP Service Account Key in JSON or Base64 formats.

In the Part 1 we’v created the service account devops-gke-nodes, but it has too many permissions and mostly designed for the GKE cluster creation. Therefore, let’s separate service accounts and create another one for deployments only.

Let’s name this service account devops-gke-deployments. We would have one for managing nodes another for deployments.

gcloud iam service-accounts create devops-gke-deployments \  --project=$PROJECT \  --description="DevOps GKE Deployments Service Account" \  --display-name="DevOps Deployments SA"

In this account we should add permissions to access Artifact Registry to download docker images that will use later for the application deployments.

PROJECT=devops-example-projectGKE_DEPLOYMENTS_SA=devops-gke-deploymentsgcloud projects add-iam-policy-binding $PROJECT \  --member="serviceAccount:$GKE_DEPLOYMENTS_SA@$PROJECT.iam.gserviceaccount.com" \  --role="roles/artifactregistry.writer"gcloud projects add-iam-policy-binding $PROJECT \  --member="serviceAccount:$GKE_DEPLOYMENTS_SA@$PROJECT.iam.gserviceaccount.com" \  --role="roles/storage.admin"

And for the terraform operations we should add those permissions:

gcloud projects add-iam-policy-binding $PROJECT \  --member="serviceAccount:$GKE_DEPLOYMENTS_SA@$PROJECT.iam.gserviceaccount.com" \  --role="roles/container.developer"gcloud projects add-iam-policy-binding $PROJECT \  --member="serviceAccount:$GKE_DEPLOYMENTS_SA@$PROJECT.iam.gserviceaccount.com" \  --role="roles/container.clusterViewer"gcloud projects add-iam-policy-binding $PROJECT \  --member="serviceAccount:$GKE_DEPLOYMENTS_SA@$PROJECT.iam.gserviceaccount.com" \  --role="roles/storage.admin"gcloud projects add-iam-policy-binding $PROJECT \  --member="serviceAccount:$GKE_DEPLOYMENTS_SA@$PROJECT.iam.gserviceaccount.com" \  --role="roles/viewer"

If you do something outside of those permissions, no worries, you can add more later.

Let’s also create Artifact Repository for the docker images

gcloud artifacts repositories create devops-docker-repo \  --repository-format=docker \  --location=us-central1 \  --description="Docker repository for DevOps"

New we are ready to download deployments SA key from GCP Web Console -> Service Account -> devops-gke-deployments and store it in repo or org level:

1. GitHub -> Source Code Repo -> Settings -> Secrets and variables -> Actions -> Repository secrets.
2. GitHub -> Organization > Settings -> Secret and variables -> Actions -> Organization secrets.

If you have only one GKE cluster, it is a good choice to store in org level.

You could have different GKE clusters for example for dev, staging, prod. In this case you will have three workflows that knows what cluster you are updating and use appropriate GCP_DEPLOYMENTS_[ENV]_SA_KEY. My recommendation is to keep association main -> prod, dev -> dev, releases/* -> staging. Do branch for each release like releases*/*yyyy-mm-dd, since you can not add more commits to tags in case of urgent prod bugs.

Here it is, we do not need manually initialize terraform and apply changes anymore. Every time when we commit to the main branch the GitHub actions automatically runs action that authenticates in GKE cluster with service account devops-gke-deployments and applies terraform scripts for ingress folder. This process called automated CI/CD pipeline for DevOps.

In your source code project

git add .git commit -m "my changes"git push

It will trigger the workflow and in the GitHub page of the repository in the tab Actions you would have one Action started to deploy the changes to the GKE cluster directly.

Summary

In this part we created terraform scripts that install ingress controller and cert manager, assign the external static IP address. Automated deployment by CI/CD pipeline through GitHub Actions.

The next is Part 3.