# LotusCloud Control Plane Architecture

## Overview

The Control Plane is the brain of LotusCloud. It manages all cloud resources, enforces policies, handles billing, and orchestrates provisioning across the Data Plane infrastructure.

Design follows AWS/GCP internal control plane patterns.

## Control Plane Services (~40 services)

### Core Services

```
api-gateway           → Request routing, auth, rate limiting, circuit breaker
iam-service           → Users, roles, policies, service accounts, OIDC
resource-manager      → Central resource registry, lifecycle management
scheduler-service     → Placement engine, node selection, zone balancing
provisioning-engine   → Multi-step resource creation/deletion
workflow-engine       → Choreograph complex provisioning workflows
billing-service       → Usage tracking, invoicing, cost analysis
quota-service         → Resource limit enforcement
audit-service         → Activity logging, compliance (ISO 27001)
policy-service        → Guardrails, compliance rules
notification-service  → Email, SMS, webhook alerting
config-service        → Feature flags, dynamic configuration
event-bus-service     → Kafka/NATS event routing
service-catalog       → Registry of all available cloud services
```

### Identity Domain

```
auth-service          → Authentication, token issuance
token-service         → JWT lifecycle, refresh tokens
user-service          → User CRUD
group-service         → Group management
role-service          → Role definitions
policy-service        → Policy documents (IAM policies)
access-key-service    → API key management
service-account-svc   → Workload identity
oidc-service          → OpenID Connect provider
```

### Organization Domain

```
organization-service  → Multi-tenant org management
project-service       → Project isolation (like GCP Projects)
folder-service        → Org hierarchy (optional)
quota-service         → Per-project and per-org quotas
tagging-service       → Resource tagging and labeling
```

## Control Plane Request Flow

```
User (CLI / Console / SDK)
          │
          ▼
    API Gateway
    ├── Rate limiting (token bucket)
    ├── Authentication (JWT validation)
    ├── Request routing
    └── API versioning (/v1, /v2)
          │
          ▼
    IAM Authorization
    ├── Check user identity
    ├── Evaluate IAM policies
    ├── Check resource-level permissions
    └── Fail-closed (deny by default)
          │
          ▼
    Resource Manager
    ├── Validate request schema
    ├── Check quota availability
    ├── Create resource record (PENDING state)
    └── Generate LRN (Lotus Resource Name)
          │
          ▼
    Scheduler
    ├── Filter eligible nodes/zones
    ├── Score candidates
    └── Select best placement
          │
          ▼
    Provisioning Engine
    ├── Create workflow
    ├── Execute steps
    ├── Handle failures/rollback
    └── Update resource state
          │
          ▼
    Event Bus (Kafka/RabbitMQ)
          │
          ├── billing-service (track cost)
          ├── audit-service (log activity)
          ├── monitoring-service (metrics)
          └── notification-service (alert)
```

## Provisioning Engine Workflows

### VM Provisioning

```
Step 1: Validate request (CPU, RAM, image, zone)
Step 2: Check quota
Step 3: Allocate subnet IP (IPAM)
Step 4: Allocate storage volume
Step 5: Schedule to compute node
Step 6: Create VM via hypervisor (KVM/Nova)
Step 7: Attach network interface
Step 8: Attach storage volume
Step 9: Update resource state → RUNNING
Step 10: Emit event vm.created
```

### Kubernetes Cluster Provisioning

```
Step 1: Validate cluster config (version, node count, CNI)
Step 2: Check quota
Step 3: Allocate network (VPC, subnet)
Step 4: Provision control plane nodes
Step 5: Bootstrap kubeadm
Step 6: Install CNI (Cilium)
Step 7: Install ingress controller
Step 8: Provision worker nodes
Step 9: Register cluster endpoint
Step 10: Return kubeconfig
Step 11: Emit event cluster.created
```

### DBaaS Provisioning

```
Step 1: Validate DB config (engine, version, storage, HA)
Step 2: Check quota
Step 3: Allocate VM/container
Step 4: Attach storage volume
Step 5: Install database engine
Step 6: Configure replication (if HA)
Step 7: Configure PgBouncer/ProxySQL
Step 8: Configure automated backup
Step 9: Register endpoint
Step 10: Emit event database.created
```

## Workflow Engine

Complex provisioning uses a workflow engine for reliability.

### Technology Options

- Temporal
- Cadence
- Argo Workflows
- Custom state machine

### Workflow Properties

- Durable execution (survives service restarts)
- Automatic retry with backoff
- Compensation (rollback on failure)
- Audit trail of every step
- Timeout handling

## Scheduler Engine

The scheduler is the brain of resource placement.

### Scheduler Pipeline

```
Request → Filter → Score → Select
```

### Filter Step

Eliminates nodes that cannot host the resource:

```
- Enough CPU available
- Enough RAM available
- Enough disk available
- Correct availability zone
- Correct region
- Node healthy
- Network capacity available
- GPU available (if required)
```

### Score Step

Ranks remaining candidates:

```
score = cpu_weight * cpu_score
      + ram_weight * ram_score
      + zone_weight * zone_balance_score
      + load_weight * current_load_score
```

### Scheduling Strategies

| Strategy | Description |
|---------|------------|
| Bin Packing | Pack resources densely to minimize nodes |
| Spread | Distribute evenly across zones |
| Least Loaded | Choose node with least utilization |
| Anti-affinity | Avoid co-locating same-tenant workloads |

### Placement Example

```
VM Request: 4 CPU, 8GB RAM, zone: auto

Filter:
  Node-01 (zone-a): 32 free CPU, 64GB free RAM → PASS
  Node-02 (zone-b): 2 free CPU → FAIL (insufficient CPU)
  Node-03 (zone-a): 16 free CPU, 32GB free RAM → PASS
  Node-04 (zone-c): 24 free CPU, 48GB free RAM → PASS

Score (spread strategy):
  Node-01: zone-a (2 existing VMs) → score 70
  Node-03: zone-a (2 existing VMs) → score 65
  Node-04: zone-c (0 existing VMs) → score 95

Selected: Node-04 (zone-c) — best zone spread
```

## Global Control Plane (Multi-Region)

```
Global Control Plane
       │
       ├── Region: VN-South Control Plane
       │      ├── Zone: vn-south-1a
       │      └── Zone: vn-south-1b
       │
       └── Region: VN-North Control Plane
              ├── Zone: vn-north-1a
              └── Zone: vn-north-1b
```

### Global Services

- Global API Gateway (geo-routed)
- Global IAM (replicated)
- Global Resource Registry
- Global DNS
- Global Load Balancer

### Regional Services

- Regional Scheduler
- Regional Provisioning Engine
- Regional Control Database
- Regional Event Bus

## Control Plane HA

### Database

- PostgreSQL HA cluster (3 nodes)
- Patroni for automatic failover
- etcd for leader election
- Synchronous replication within zone
- Async replication across zones

### Services

- All services are stateless
- Deployed on Kubernetes
- Minimum 3 replicas per service
- Horizontal Pod Autoscaler
- Pod anti-affinity across zones

### Event Bus

- Kafka cluster (3+ brokers)
- Multi-zone replication
- At-least-once delivery

## Internal Communication

| Type | Protocol | Use Case |
|------|----------|----------|
| Sync | gRPC | Service-to-service RPC |
| Sync | REST | External API |
| Async | Kafka/RabbitMQ | Event processing |
| Auth | mTLS | Service identity |
| Discovery | Kubernetes DNS | Service resolution |

## Control Plane Scaling Strategy

```
Load increases
     │
     ▼
HPA scales service replicas
     │
     ▼
If cluster full → add nodes (Cluster Autoscaler)
     │
     ▼
If region full → spill to next region
```

### Scale Targets

| Service | Min Replicas | Max Replicas |
|---------|-------------|-------------|
| api-gateway | 3 | 20 |
| iam-service | 3 | 10 |
| resource-manager | 3 | 15 |
| scheduler | 2 | 10 |
| billing-service | 2 | 10 |
| provisioning-engine | 3 | 20 |