Deploying K8S application in GKE to GCP Endpoint. Part 4.
Building the CI/CD pipeline based on Terraform to deploy application in GKE cluster.
In previous Part 3 we created a simple web application that is using FAST API framework in python to expose REST API to public.
Now it is time to deploy this application to the GKE service, build CI/CD pipeline as expose it as a public GCP Endpoint.
Let’s start by adding the tool file write_openapi.pythat would generate openapi.yaml
import jsonimport yamlfrom copy import deepcopyfrom fastapi.openapi.utils import get_openapifrom config.config import project_id, secret_namedef convert_v3_to_v2(openapi_v3: dict) -> dict: """ Minimal conversion from OpenAPI v3 to v2 spec. Supports: - info - host (added separately) - basePath ("/") - schemes (["https", "http"]) - paths with GET/POST and simple response schemas (type, properties) """ openapi_v2 = {} # Basic top-level fields openapi_v2["swagger"] = "2.0" openapi_v2["info"] = openapi_v3.get("info", {}) # Convert paths paths_v3 = openapi_v3.get("paths", {}) paths_v2 = {} for path, methods in paths_v3.items(): paths_v2[path] = {} for method, details in methods.items(): op = {} op["summary"] = details.get("summary", "") op["operationId"] = details.get("operationId", None) op["produces"] = ["application/json"] # Parameters (query/path/body) parameters = [] for param in details.get("parameters", []): param_v2 = deepcopy(param) # OpenAPI v3 uses "schema" for body/query params, v2 uses "type" if "schema" in param_v2: schema = param_v2.pop("schema") if "type" in schema: param_v2["type"] = schema["type"] parameters.append(param_v2) op["parameters"] = parameters # Responses: convert content->schema to v2 schema responses = {} for code, resp in details.get("responses", {}).items(): resp_v2 = {} description = resp.get("description", "") resp_v2["description"] = description # Convert OpenAPI v3 response content to v2 schema content = resp.get("content", {}) if "application/json" in content: schema = content["application/json"].get("schema", None) if schema: resp_v2["schema"] = schema responses[code] = resp_v2 op["responses"] = responses paths_v2[path][method] = op openapi_v2["paths"] = paths_v2 # Remove servers (not in v2) if "servers" in openapi_v3: del openapi_v3["servers"] return openapi_v2def generate_openapi_yaml(host, target, output_file): """Generate OpenAPI schema in YAML format for GCP Cloud Endpoints.""" from app import app schema = get_openapi( title=f"{service_name} API HTTP Service", version="1.0.0", description=f"{service_name} API HTTP Service for GKE service", routes=app.routes, ) # Manually include /docs and /openapi.json schema.setdefault("paths", {}) schema["paths"]["/docs"] = { "get": { "summary": "Swagger UI", "description": "Swagger UI for this API", "operationId": "getDocs", "responses": {"200": {"description": "Swagger UI"}}, } } schema["paths"]["/openapi.json"] = { "get": { "summary": "OpenAPI schema", "description": "OpenAPI JSON for this API", "operationId": "getOpenapiSchema", "responses": {"200": {"description": "OpenAPI schema"}}, } } openapi_schema = convert_v3_to_v2(schema) openapi_schema["swagger"] = "2.0" openapi_schema["host"] = host openapi_schema["basePath"] = "/" openapi_schema["schemes"] = ["https", "http"] openapi_schema["x-google-endpoints"] = [ { "name": host, "target": target, } ] with open(output_file, "w") as f: yaml.dump(openapi_schema, f, sort_keys=False) print("✅ OpenAPI schema written to openapi.yaml")# GCP Cloud Endpoints configurationGCP_SERVICE_HOST = f"{service_name}.endpoints.{project_id}.cloud.goog"OUTPUT_FILE = "openapi.yaml"def main(): parser = argparse.ArgumentParser(description="FastAPI app or OpenAPI generator for GCP") parser.add_argument("--host", default=GCP_SERVICE_HOST, help="Host to run FastAPI server on") parser.add_argument("--target", help="External IP of the GCP Endpoint") parser.add_argument("--output", default=OUTPUT_FILE, help="Output file of the schema") args = parser.parse_args() generate_openapi_yaml(host=args.host, target=args.target, output_file=args.output) sys.exit(0)if __name__ == "__main__": main()This script will ask FAST API to provide all known routes, and additionally adds specific for FAST API routes like /docs/ and /openapi.json to the schema, adds external ip address that we got in the Part 2 and writes to the file openapi.yaml
swagger: '2.0'info: title: example-api-service API HTTP Service description: example-api-service API HTTP Service for GKE service version: 1.0.0host: example-api-service.endpoints.devops-example-project.cloud.googbasePath: /schemes:- https- httpx-google-endpoints:- name: example-api-service.endpoints.devops-example-project.cloud.goog target: <your external static ip>paths: /: get: summary: Read Root operationId: read_root__get produces: - application/json parameters: [] responses: '200': description: Successful Response /v1/status: get: summary: Status operationId: status_v1_status_get produces: - application/json parameters: [] responses: '200': description: Successful Response /docs: get: summary: Swagger UI operationId: getDocs produces: - application/json parameters: [] responses: '200': description: Swagger UI /openapi.json: get: summary: OpenAPI schema operationId: getOpenapiSchema produces: - application/json parameters: [] responses: '200': description: OpenAPI schemaNow you have a tool script in the project that can generate schema after the API changes automatically.
Let’s register it in GCP Endpoints, so it would create an endpoint for the first time call or update it for sequential calls.
gcloud endpoints services deploy openapi.yaml --project=devops-example-projectIn GCP Web interface in ‘Endpoints’ section you can see that endpoint was added with declared interface.
Now there is the time to create additional GCP Service Account that we are going to use to run this first application in GKE.
PROJECT=devops-example-projectGKE_SERVICES_SA=devops-gke-servicesgcloud iam service-accounts create $GKE_SERVICES_SA \ --project=$PROJECT \ --description="API Services GKE Service Account" \ --display-name="API Services GKE SA"gcloud projects add-iam-policy-binding $PROJECT \ --member="serviceAccount:$GKE_SERVICES_SA@$PROJECT.iam.gserviceaccount.com" \ --role="roles/servicemanagement.serviceController"gcloud secrets add-iam-policy-binding $PROJECT \ --member="serviceAccount:$GKE_SERVICES_SA@$PROJECT.iam.gserviceaccount.com" \ --role="roles/secretmanager.secretAccessor"Let’s open GCP Web Interface, go to the IAM&Admin and find thie Service Account devops-gke-services, copy the JSON content and convert it to Base⁶⁴. We would use this service account as GCP_SERVICES_SA_KEY in our terraform and workflow scripts. Together with runtime GCP_SERVICES_SA_KEY do not forget to add GCP_DEPLOYMENTS_SA_KEY that we created in Part 2.
Let’s add it to the repository in GitHub as Base64 value in 1 or in 2:
- GitHub -> Source Code Repo -> Settings -> Secrets and variables -> Actions -> Repository secrets.
- GitHub -> Organization > Settings -> Secret and variables -> Actions -> Organization secrets.
Now, we have compled registration ot GCP endpoint, GCP Service Account and GitHub reference on Service Account.
Terraform
For the next step we would need ready terraform scripts.
All scripts are located in terraform folder in source code repository.
Let’s create .terraform-version file.
1.12.1Now we can define variables for the service.
variable "project_id" { type = string}variable "region" { type = string}variable "region_with_zone" { type = string}variable "cluster_name" { type = string}variable "image_name" { type = string}variable "image_tag" { type = string default = "latest"}variable "repo_name" { type = string}variable "service_account_key" { type = string sensitive = true}With predefined parameters in terraform.tfvars file
image_name = "example-api-service"project_id = "devops-example-project"region = "us-central1"region_with_zone = "us-central1-a"repo_name = "devops-docker-repo"# do not put here service_account_keyFrom those parameters we are going to change only image_name if we create another project.
Let’s create backend.tf file
terraform { backend "gcs" { bucket = "devops-terraform-state" # You must create this bucket first prefix = "devops-cluster-dev/example-api-service" }}Now we are ready to add main.tf file that created major resources
provider "google" { project = var.project_id region = var.region_with_zone}data "google_client_config" "default" {}data "google_container_cluster" "gke" { name = var.cluster_name location = var.region_with_zone}provider "kubernetes" { host = "https://${data.google_container_cluster.gke.endpoint}" cluster_ca_certificate = base64decode(data.google_container_cluster.gke.master_auth[0].cluster_ca_certificate) token = data.google_client_config.default.access_token}resource "kubernetes_namespace" "current" { metadata { name = "${var.image_name}" } lifecycle { ignore_changes = [metadata] }}Here we are not creating gke cluster, but using exiting one defined in Part 1.
The principal idea is to create a separate namespace for each deployment of service or application, that what we are doing here in kubernetes_namespace.current . In case if something significantly went wrong in kubectl we can delete the namespace of the application and re-deploy it again, especially if it is a stateless deployment.
Now it is the time to define our primary service.tf file.
resource "kubernetes_deployment" "api_deployment" { metadata { name = var.image_name namespace = kubernetes_namespace.current.metadata[0].name labels = { app = var.image_name } } spec { replicas = 1 selector { match_labels = { app = var.image_name } } template { metadata { labels = { app = var.image_name } } spec { container { name = "${var.image_name}" image = "${var.region}-docker.pkg.dev/${var.project_id}/${var.repo_name}/${var.image_name}:${var.image_tag}" port { container_port = 8000 } resources { limits = { cpu = "250m" memory = "512Mi" } requests = { cpu = "100m" memory = "256Mi" } } readiness_probe { http_get { path = "/" port = 8000 } initial_delay_seconds = 2 period_seconds = 5 } liveness_probe { http_get { path = "/" port = 8000 } initial_delay_seconds = 5 period_seconds = 10 } env { name = "PORT" value = "8000" } } } } } depends_on = [kubernetes_namespace.current]}resource "kubernetes_service" "api_service" { metadata { name = var.image_name namespace = kubernetes_namespace.current.metadata[0].name } spec { selector = { app = var.image_name } port { port = 80 target_port = 8000 protocol = "TCP" } type = "ClusterIP" } depends_on = [kubernetes_namespace.current]}The FAST API and Dockerfile are exporting 8000 port, not the 80 port, therefore in out api_service we need to redirect those ports. This file also contains the api_deployment that defines the actual service running on 8000 port and using image that we built earlier. Resources section could be used to increase/decrease resources of the particular pod. The two sections readiness_probe and liveness_probe guarantee that Terraform will go to the next step only after ensuring that root path is working, that is required by GCP/GKE, that’s why we have root endpoint in our service.
The next file we need is proxy.tf . This file is responsible to work with GCP Endpoint and track all calls of the methods that we automatically declared in openapi.xml . The proxy name is esp-v2. First we forward all traffic to proxy and proxy itself forward traffic to the service.
# usually nothing to change here by @Alexresource "kubernetes_deployment" "proxy_deployment" { metadata { name = "${var.image_name}-proxy" namespace = kubernetes_namespace.current.metadata[0].name labels = { app = "${var.image_name}-proxy" } } spec { replicas = 1 selector { match_labels = { app = "${var.image_name}-proxy" } } template { metadata { labels = { app = "${var.image_name}-proxy" } } spec { container { name = "${var.image_name}-proxy" image = "gcr.io/endpoints-release/endpoints-runtime:2" args = [ "--listener_port=8080", "--backend=${kubernetes_service.api_service.metadata[0].name}:80", "--service=${var.image_name}.endpoints.${var.project_id}.cloud.goog", "--rollout_strategy=managed", ] port { container_port = 8080 } resources { limits = { cpu = "500m" memory = "512Mi" } requests = { cpu = "100m" memory = "128Mi" } } readiness_probe { http_get { path = "/" port = 8080 } initial_delay_seconds = 2 period_seconds = 5 } liveness_probe { http_get { path = "/" port = 8080 } initial_delay_seconds = 5 period_seconds = 10 } } } } } depends_on = [kubernetes_namespace.current, kubernetes_service.api_service]}resource "kubernetes_service" "proxy_service" { metadata { name = "${var.image_name}-proxy" namespace = kubernetes_namespace.current.metadata[0].name } spec { selector = { app = "${var.image_name}-proxy" } port { port = 80 target_port = 8080 protocol = "TCP" } type = "ClusterIP" } depends_on = [kubernetes_namespace.current]}It is important to provide the correct domain name of the GCP Endpoint to this proxy and correct port to the service. I kept it simple and using in all services 80 port.
In order to serve the traffic we need to create ingress resource and define what host we are going to listen.
Let’s create the ingress.tf file.
resource "kubernetes_manifest" "ingress" { manifest = { apiVersion = "networking.k8s.io/v1" kind = "Ingress" metadata = { name = "${var.image_name}-ingress" namespace = kubernetes_namespace.current.metadata[0].name annotations = { "cert-manager.io/cluster-issuer" = "letsencrypt-prod" "cert-manager.io/issue-temporary-certificate" = "true" "cert-manager.io/acme-challenge-type" = "http01" } } spec = { ingressClassName = "nginx" rules = [ { host = "${var.image_name}.endpoints.${var.project_id}.cloud.goog" http = { paths = [ { path = "/" pathType = "Prefix" backend = { service = { name = kubernetes_service.proxy_service.metadata[0].name port = { number = 80 } } } } ] } }, ] tls = [ { hosts = ["${var.image_name}.endpoints.${var.project_id}.cloud.goog"] secretName = "${var.image_name}-tls" } ] } } field_manager { name = "terraform" force_conflicts = true } depends_on = [kubernetes_namespace.current, kubernetes_service.proxy_service]}In this file we forward all traffic to the proxy_service that we defined earlier. Also we are declaring the serverName for the TLS certificate that would be generated automatically since we have special annotations for that.
And finally we need outputs.tf file to display terraform variables.
output "namespace" { description = "Namespace where the application is deployed" value = kubernetes_namespace.current.metadata[0].name}output "image_used" { description = "Docker image deployed" value = "${var.region}-docker.pkg.dev/${var.project_id}/${var.repo_name}/${var.image_name}:${var.image_tag}"}We want to know that namespace was used and what image was deployed.
CI/CD
Now we have fully developed application and terraform scripts, created GCP endpoint and service account.
Let’s create the file .github/workflows/deploy-gke.yaml
name: Build and Push to GCPon: push: branches: - main env: IMAGE_NAME: example-api-service PROJECT_ID: devops-example-project REGION: us-central1 REGION_WITH_ZONE: us-central1-a REPOSITORY: docker.pkg.dev REPO_NAME: devops-docker-repo CLUSTER_NAME: devops-clusterjobs: build-and-push: runs-on: ubuntu-latest outputs: image_tag: ${{ github.sha }} steps: - name: Checkout source code uses: actions/checkout@v3 - name: Set up Docker uses: docker/setup-buildx-action@v3 - name: Authenticate to Google Cloud uses: google-github-actions/auth@v2 with: credentials_json: '${{ secrets.GCP_DEPLOYMENTS_SA_KEY }}' - name: Configure Docker for Artifact Registry run: | gcloud auth configure-docker ${{ env.REGION }}-${{ env.REPOSITORY }} --quiet - name: Build and Push Docker image run: | IMAGE_SHA=${{ env.REGION }}-${{ env.REPOSITORY }}/${{ env.PROJECT_ID }}/${{ env.REPO_NAME }}/${{ env.IMAGE_NAME }}:${{ github.sha }} IMAGE_LATEST=${{ env.REGION }}-${{ env.REPOSITORY }}/${{ env.PROJECT_ID }}/${{ env.REPO_NAME }}/${{ env.IMAGE_NAME }}:latest docker build -t $IMAGE_SHA -t $IMAGE_LATEST -f Dockerfile.dev . docker push $IMAGE_SHA docker push $IMAGE_LATEST echo "Pushed images:" echo " - $IMAGE_SHA" echo " - $IMAGE_LATEST" terraform-deploy: needs: build-and-push runs-on: ubuntu-latest defaults: run: working-directory: terraform steps: - name: Checkout source code uses: actions/checkout@v3 - name: Set up Terraform uses: hashicorp/setup-terraform@v2 with: terraform_version: 1.12.1 - name: Authenticate to Google Cloud uses: google-github-actions/auth@v2 with: credentials_json: '${{ secrets.GCP_DEPLOYMENTS_SA_KEY }}' - name: Get GKE Credentials run: | gcloud container clusters get-credentials ${{ env.CLUSTER_NAME }} \ --region ${{ env.REGION_WITH_ZONE }} \ --project ${{ env.PROJECT_ID }} - name: Terraform Init run: terraform init - name: Terraform Plan run: terraform plan -var="image_tag=${{ needs.build-and-push.outputs.image_tag }}" working-directory: terraform env: TF_VAR_service_account_key: ${{ secrets.GCP_SERVICES_SA_KEY }} - name: Terraform Apply run: terraform apply -auto-approve -var="image_tag=${{ needs.build-and-push.outputs.image_tag }}" working-directory: terraform env: TF_VAR_service_account_key: ${{ secrets.GCP_SERVICES_SA_KEY }}The CI/CD pipeline successfully created and after each commit you will have new service deployed in GKE cluster by terraform.
Summary
In this Part we finished the set of articles that are using Terraform for GKE cluster creation and automation with GitHub workflows.