Cloud Task Engine: GCP Microservices Platform

This project is a serverless microservices architecture deployed to Google Cloud Platform (GCP). It leverages Node.js for the API Gateway and Worker services, and Go for the Scheduler service. Infrastructure provisioning is managed via Terraform (IaC), and automated CI/CD deployment is handled by GitHub Actions. The system is designed to run entirely for free under the GCP Free Tier limit.

System Screenshots

Dashboard Upper Section

Dashboard Lower Section

GCP Firestore Collections

---

System Architecture

┌──────────────────────────────────────────────────────────────┐
│                         Developer                            │
│                      git push main                           │
│                      └──────┬────────────────────────────────┘
│                             │
│                             ▼
│ ┌──────────────────────────────────────────────────────────┐ │
│ │                    GitHub Actions                        │ │
│ │  ┌─────────────┐   ┌──────────────┐   ┌────────────────┐ │ │
│ │  │  Terraform  │──>│ Docker Build │──>│  Deploy Cloud  │ │ │
│ │  │   Apply     │   │ & Push Image │   │      Run       │ │ │
│ │  └─────────────┘   └──────────────┘   └────────────────┘ │ │
│ └───────────────────────────┬──────────────────────────────┘ │
└─────────────────────────────┼────────────────────────────────┘
                              │
                              ▼
┌──────────────────────────────────────────────────────────────┐
│                      GCP Project                             │
│                                                              │
│   ┌─────────────────────────────────────────────────────┐   │
│   │              Artifact Registry                      │   │
│   │         (Stores Docker Images)                      │   │
│   └─────────────────────────────────────────────────────┘   │
│                                                              │
│   ┌──────────────┐                     ┌──────────────────┐  │
│   │ API Gateway  │                     │    Scheduler     │  │
│   │  Cloud Run   │                     │   Cloud Run      │  │
│   │  (Node.js)   │                     │      (Go)        │  │
│   │              │                     │                  │  │
│   │ CRUD items   │                     │ Manages schedule │  │
│   └──────┬───────┘                     └────────┬─────────┘  │
│          │                                      │            │
│          │                                      ▼            │
│          │                             ┌──────────────────┐  │
│          │                             │   Cloud Tasks    │  │
│          │                             │     Queue        │  │
│          │                             └────────┬─────────┘  │
│          │                                      │            │
│          │                                      ▼            │
│          │                             ┌──────────────────┐  │
│          │                             │    Worker        │  │
│          │                             │   Cloud Run      │  │
│          │                             │   (Node.js)      │  │
│          │                             │                  │  │
│          │                             │ Processes job    │  │
│          │                             └────────┬─────────┘  │
│          │                                      │            │
│          └─────────────────┬────────────────────┘            │
│                            │                                 │
│                            ▼                                 │
│                  ┌─────────────────┐                         │
│                  │    Firestore    │                         │
│                  │                 │                         │
│                  │  /items         │                         │
│                  │  /job_results   │                         │
│                  │  /scheduled_jobs│                         │
│                  └─────────────────┘                         │
└──────────────────────────────────────────────────────────────┘

All Cloud Run services are configured with min_instances = 0 (scale to zero), meaning they scale down to zero instances when idle to prevent incurring any billing costs.

Service Breakdown

• API Gateway (Node.js): The main entry point. Receives external HTTP requests and performs CRUD operations on the items collection in Firestore (Local Port: 3000).
• Worker (Node.js): Asynchronously processes heavy background jobs and saves the results in the Firestore job_results collection (Local Port: 3001). This service is private and only accessible by authorized service accounts (like Google Cloud Tasks).
• Scheduler (Go): Stores scheduled tasks in the Firestore scheduled_jobs collection. When triggered periodically via the /run endpoint, it queries due tasks and dispatches them to the Google Cloud Tasks Queue using OIDC tokens for secure and reliable job execution (Local Port: 3002).

Project Directory Structure

cloud-task-engine/
│
├── README.md
├── start.bat                   Windows batch script to run all services locally
├── start_services.ps1          PowerShell helper script for local execution
├── key.json                    GCP Service Account credentials file (gitignored)
│
├── terraform/                  Infrastructure as Code (IaC) configuration
│   ├── main.tf                 Main module connector
│   ├── variables.tf            Input variable definitions
│   ├── outputs.tf              Cloud Run output URLs
│   └── modules/                Sub-modules (Cloud Run, Firestore, IAM, Registry)
│
├── services/                   Application Source Code
│   ├── api-gateway/            API Gateway Service (Node.js) + `.env.example`
│   ├── worker/                 Background Worker Service (Node.js) + `.env.example`
│   └── scheduler/              Scheduler Service (Go) + `.env.example`
│
└── .github/
    └── workflows/
        └── deploy.yml          CI/CD automated deployment workflow to GCP

---

GCP Deployment Guide

Follow these step-by-step instructions to deploy the entire microservices stack to your Google Cloud Platform account.

Step 1: Prepare GCP Project

1. Log in to your GCP account via terminal using the gcloud CLI.
2. Create a new GCP project using gcloud projects create YOUR-PROJECT-ID.
3. Set your active project using gcloud config set project YOUR-PROJECT-ID.
4. Enable billing for the project in the GCP Console to allow resource provisioning.

Step 2: Provision Infrastructure with Terraform

1. Navigate to the terraform directory in your workspace.
2. Copy the variables template using cp terraform.tfvars.example terraform.tfvars.
3. Edit the terraform.tfvars file and specify your configuration (enter project_id, region, and environment).
4. Initialize Terraform: terraform init.
5. Provision resources: terraform apply (this creates Firestore, Artifact Registry, IAM roles, and sets up Cloud Run placeholder services).

Step 3: Setup GitHub Actions CI/CD Secrets

Deployments are automated on git pushes via GitHub Actions using Workload Identity Federation (WIF) for keyless GCP authentication.

1. Create and configure a Workload Identity Pool in your GCP project using gcloud CLI.
2. Copy the resulting WIF provider resource path.
3. Open your GitHub repository, go to Settings -> Secrets and variables -> Actions.
4. Add the following repository secrets:
   • GCP_PROJECT_ID: Your GCP Project ID.
   • WIF_PROVIDER: Your WIF provider resource path.
   • WIF_SERVICE_ACCOUNT: The email address of the GitHub Actions service account created by Terraform (e.g. github-actions-sa@your-project-id.iam.gserviceaccount.com).

Step 4: Push and Deploy

Commit and push your changes to your main branch to trigger the automated CI/CD deployment:

git add .
git commit -m "feat: complete configuration and deploy infrastructure"
git push origin main

Track the build and deployment progress under the Actions tab of your GitHub repository.

Step 5: Retrieve GCP Cloud Run Service URLs

After the GitHub Actions workflow successfully deploys the microservices, retrieve the public HTTPS URLs of the Cloud Run services.

1. Navigate to the terraform directory in your terminal:

   cd terraform

2. Run the following command to print the service URLs:

   terraform output

   This will output the HTTPS endpoints:
   • api_gateway_url: The URL for the Node.js API Gateway.
   • scheduler_url: The URL for the Go Scheduler.
   • worker_url: The URL for the private Node.js Worker.

Step 6: Testing the Microservices on GCP

Testing on GCP differs from local testing. Because Cloud Run manages the routing, there are no port numbers (services communicate securely over standard port 443 HTTPS).

1. Verify Health Status (Public Services):

   • Open a browser or use Postman to send a GET request to [api_gateway_url]/health (should return { "status": "ok", "service": "api-gateway" }).
   • Send a GET request to [scheduler_url]/health (should return { "service": "scheduler", "status": "ok" }).

2. Verify Worker Private Status:

   • Send a GET request to [worker_url]/health or [worker_url]/results.
   • This should return a 403 Forbidden error. This is the expected security behavior: the Worker Cloud Run service is private (no-allow-unauthenticated) and is only accessible by Google Cloud Tasks queues or authorized service accounts using OIDC tokens.

3. Schedule and Trigger Tasks:

   • Send a POST request to [scheduler_url]/schedule with JSON body:

     {
       "name": "gcp-sync-test",
       "run_at": "2026-01-01T00:00:00Z",
       "payload": {
         "cloud_mode": true
       }
     }

     This should return 201 Created indicating the task document is stored in Firestore with status pending.
   • Send a POST request to [scheduler_url]/run to trigger task execution. The Scheduler service will read the pending task and dispatch it to the Google Cloud Tasks Queue, which securely invokes the private Worker via OIDC.
   • Verify the task execution details by inspecting the collections (scheduled_jobs and job_results) inside the GCP Firestore console.

Step 7: Connecting the Dashboard to Deployed GCP Services

To use the frontend dashboard with your live GCP Cloud Run environment:

1. Open dashboard/dashboard.js and update the apiBase variable under the state configuration to point to your public scheduler_url:

   const state = {
     theme: 'theme-glass-bento',
     viewState: 'nominal',
     dataSource: 'live',
     apiBase: 'https://scheduler-xxxxxx.a.run.app', // Replace with your scheduler_url
     ...
   };

2. Start the local dashboard server:

   node dashboard/serve.js

3. Open http://localhost:3050 in your web browser. The dashboard will now communicate with the GCP Firestore database through the deployed Go Scheduler Cloud Run instance.

---

Local Setup & Installation

There are several ways to spin up the Firestore database and run the microservices locally. Choose the setup option that best fits your local environment.

Main Prerequisites

• Node.js v18 or newer
• Go v1.21 or newer

Option A: Connecting directly to GCP Firestore (Using key.json)

This option is recommended if you want to test database writes/reads against your real GCP Firestore instance from your local computer.

1. Go to GCP Console -> IAM & Admin -> Service Accounts.
2. Select your service account (e.g. cloud-run-sa or create a new one with the Cloud Datastore User role).
3. Click the Keys tab -> Add Key -> Create new key -> select JSON.
4. Save the downloaded credential file as key.json inside the root folder of this project.
5. Install service dependencies:

   cd services/api-gateway && npm install && cd ../..
   cd services/worker && npm install && cd ../..
   cd services/scheduler && go mod tidy && cd ../..

6. Spin up all services automatically by double-clicking the start.bat file in the root folder, or running this in PowerShell:

   .\start.bat

Option B: Using Google Cloud SDK (Firestore Emulator)

A completely offline and free option utilizing the built-in emulator from the Google Cloud CLI.

1. Ensure the gcloud CLI is installed on your computer.
2. Install the emulator component:

   gcloud components install cloud-firestore-emulator

3. Start the local emulator:

   gcloud emulators firestore start --host-port=localhost:8090

4. In separate terminal windows, launch each service with the emulator host environment variable:
   • API Gateway:

     cd services/api-gateway
     $env:FIRESTORE_EMULATOR_HOST="localhost:8090"
     npm.cmd start

   • Worker:

     cd services/worker
     $env:FIRESTORE_EMULATOR_HOST="localhost:8090"
     npm.cmd start

   • Scheduler:

     cd services/scheduler
     $env:FIRESTORE_EMULATOR_HOST="localhost:8090"
     $env:GCP_PROJECT="local-dev"
     $env:WORKER_URL="http://localhost:3001"
     # QUEUE_PATH and CLOUD_RUN_SA_EMAIL are left blank in local testing
     go run main.go

     (Note: In local testing, Go Scheduler automatically falls back to direct HTTP calls to the Worker since Google Cloud Tasks is a cloud-only resource. This keeps local development 100% offline and free).

Option C: Using Docker (Firestore Emulator)

An offline option if you do not want to install any local SDK binaries other than Docker.

1. Ensure Docker Desktop is running on your machine.
2. Spin up the official Google SDK container running the emulator:

   docker run -d --name firestore-emulator -p 8090:8090 google/cloud-sdk:alpine gcloud beta emulators firestore start --host-port=0.0.0.0:8090

3. Connect your local services to the Docker container by setting the environment variable $env:FIRESTORE_EMULATOR_HOST="localhost:8090" before starting each process (refer to Option B for start commands).

Option D: Using Node.js / firebase-tools (Firestore Emulator)

An offline option if you want to avoid installing the Google Cloud CLI or Docker.

1. Ensure Java v11 or newer is installed on your computer.
2. Run the Firebase emulator suite directly using npx:

   npx firebase-tools emulators:start --only firestore

3. Connect your local services by setting $env:FIRESTORE_EMULATOR_HOST="localhost:8090" before running them (refer to Option B for start commands).

---

Dashboard User Interface

This project features a live, interactive scheduler dashboard built using pure HTML, custom Vanilla CSS, and state-driven Javascript. The dashboard provides a visual interface for managing and triggering background tasks.

Dashboard Layout

Dashboard Upper Section

The upper section of the dashboard consists of:

• Connection Status Badge: A live status indicator in the top header that shows "Live API Loop" when the dashboard is successfully polling the local Go Scheduler.
• KPI Metrics Panel: Four real-time statistics cards:
  • Total Dispatched: The cumulative count of all scheduled tasks.
  • Currently Queued: The current number of jobs in a pending or running state.
  • Success Rate: The percentage of completed tasks relative to failed ones.
  • Avg Worker Latency: The average execution latency of the background workers.
• 3D Core Visualizer: An interactive 3D core representation that accelerates based on active scheduling workloads.

Dashboard Lower Section

The lower section handles execution control and audit logging:

• Pending Jobs Queue: Lists jobs currently in a pending or running state. Users can click "Trigger /run" to manually dispatch due tasks or click the cancel button to delete a scheduled job.
• Task History Log: An interactive audit table for completed and failed job executions. It features text searching, filtering (by status and target type), sorting, pagination, and a client-side "Export CSV" tool.
• Quick Schedule Drawer: A slide-over panel to schedule new tasks. It includes fields for task name, dispatch target type, execution priority, delay interval, and a JSON payload editor.

---

API Integration and Port Design

Why the Dashboard Connects to Port 3002

The frontend dashboard interfaces directly with port 3002 (Go Scheduler).

• Port 3002 (Go Scheduler) acts as the coordinator for all task schedules. It handles the list and schedule requests (GET /jobs and POST /schedule), writing them to the scheduled_jobs collection in Firestore.
• Port 3001 (Worker) is a private processor service that executes due tasks and writes results to the job_results collection. It does not track pending schedules, so connecting the dashboard to port 3001 would make it impossible to list or manage the active queue.
• Port 3000 (API Gateway) is designed for a separate CRUD collection (/items) and is not involved in the task scheduler workflow. It is not integrated with the dashboard because the dashboard is focused exclusively on background jobs and queues.

Requirements for Running the Dashboard

To use the dashboard in live mode, you must run the backend API services (specifically Go Scheduler and Worker). If the Go Scheduler service on port 3002 is not running, the dashboard will display a connection error state and will fail to load or schedule tasks.

---

Running and Testing the Frontend

How to Install and Run

1. Start the backend services (Go Scheduler and Worker). Ensure you have configured the key.json file in the root directory.

   .\start.bat

2. Navigate to the dashboard directory and start the local static server:

   node dashboard/serve.js

3. Open http://localhost:3050 in your web browser. The dashboard will automatically connect to port 3002.

How to Test

1. Open the dashboard page. Verify that the connection badge in the header displays "Live API Loop".
2. Click New Task to open the schedule drawer. Enter a task name, select a priority and dispatch target, and click Schedule Task.
3. Confirm that the task appears in the Pending Jobs Queue card.
4. Verify in the GCP Firestore console under the scheduled_jobs collection that the new task document is written.
5. Click Trigger /run in the dashboard. Verify that the task executes, disappears from the pending queue, and is added to the Task History log as completed.

---

Postman Testing Documentation

Execute the following 8 testing steps in order inside Postman to verify the end-to-end functionality of your microservices:

1. Health Check: API Gateway

Verifies the API Gateway (Port 3000) is running locally.

• Method: GET
• Request URL: http://localhost:3000/health
• Response: 200 OK with JSON body {"status": "ok", "service": "api-gateway"}.
• Screenshot Reference:

2. Health Check: Worker

Verifies the Worker (Port 3001) is running locally.

• Method: GET
• Request URL: http://localhost:3001/health
• Response: 200 OK with JSON body {"status": "ok", "service": "worker"}.
• Screenshot Reference:

3. Health Check: Go Scheduler

Verifies the Go Scheduler (Port 3002) is running locally.

• Method: GET
• Request URL: http://localhost:3002/health
• Response: 200 OK with JSON body {"service": "scheduler", "status": "ok"}.
• Screenshot Reference:

4. Uji API Gateway: Create New Item

Creates a new document directly in your cloud Firestore instance.

• Method: POST
• Request URL: http://localhost:3000/api/items
• Headers: Content-Type: application/json
• Body (raw -> JSON):

  {
    "name": "Laptop ASUS ROG",
    "description": "Spesifikasi gaming Core i9"
  }

• Response: 201 Created with JSON body containing the generated unique Firestore document ID.
• Screenshot Reference:

5. Uji API Gateway: Fetch All Items

Retrieves all items stored in the Firestore collection.

• Method: GET
• Request URL: http://localhost:3000/api/items
• Response: 200 OK with a JSON array listing the items.
• Screenshot Reference:

6. Uji Scheduler: Schedule New Task

Registers a scheduled background job in the system.

• Method: POST
• Request URL: http://localhost:3002/schedule
• Headers: Content-Type: application/json
• Body (raw -> JSON):

  {
    "name": "send-welcome-email",
    "run_at": "2024-01-01T00:00:00Z",
    "payload": {
      "email": "user@example.com",
      "subject": "Selamat Datang!"
    }
  }

• Response: 201 Created showing the created job status as pending.
• Screenshot Reference:

7. Uji Scheduler: Trigger Job Execution (Run)

Triggers the scheduler to parse due tasks and dispatch them to the Worker.

• Method: POST
• Request URL: http://localhost:3002/run
• Response: 200 OK showing the number of executed jobs and their IDs.
• Screenshot Reference:

8. Uji Worker: Fetch Execution Results

Confirms that the Worker received the dispatched job and completed the background execution.

• Method: GET
• Request URL: http://localhost:3001/results
• Response: 200 OK listing the finished jobs with the status completed.
• Screenshot Reference: