# LotusCloud Hyperscaler Architecture

## Overview

LotusCloud is a private cloud platform designed with hyperscaler architecture patterns from AWS, GCP, and Azure. It provides IaaS + PaaS + Managed Services for enterprise workloads deployed in local/private datacenters using OpenStack + Kubernetes + Ceph + SDN.

## Architecture Layers

```
Users
   │
   ▼
Edge Network (CDN / WAF / DDoS / Anycast)
   │
   ▼
Global API Gateway
   │
   ▼
Identity & IAM
   │
   ▼
Control Plane
   │
   ├── Resource Manager
   ├── Scheduler
   ├── Provisioning Engine
   ├── Billing Engine
   ├── Quota Manager
   ├── Event Bus
   │
   ▼
Data Plane
   │
   ├── Compute Cluster (KVM / Kubernetes)
   ├── Storage Cluster (Ceph)
   ├── Network Cluster (SDN / OVN)
   │
   ▼
Physical Datacenter (Spine-Leaf / Rack / Server)
```

## Control Plane

Control plane is the brain of the cloud. All user requests flow through here.

### Core Control Plane Services

| Service | Responsibility |
|---------|---------------|
| api-gateway | Authentication, rate limiting, request routing, API versioning |
| iam-service | Users, roles, groups, policies, service accounts, access keys, OIDC |
| resource-manager | Central registry of all cloud resources |
| scheduler-service | Placement decisions, resource allocation, anti-affinity, zone balancing |
| provisioning-engine | Provision VM, K8s cluster, DB instance, storage volume, LB |
| workflow-engine | Multi-step provisioning orchestration |
| billing-service | Track CPU hours, RAM hours, storage GB, network traffic, GPU hours |
| quota-service | Prevent overcommit, enforce resource limits |
| audit-service | Log all activity: creation, deletion, permission changes, config updates |
| policy-service | Compliance enforcement, guardrails |
| notification-service | Email, SMS, webhook alerts |
| event-bus | Kafka/NATS async event distribution |

### Control Plane Request Flow

```
User Request
     │
     ▼
API Gateway
     │
     ▼
IAM Authentication + Authorization
     │
     ▼
Resource Manager (create record, validate quota)
     │
     ▼
Scheduler (choose placement)
     │
     ▼
Provisioning Engine (execute workflow)
     │
     ▼
Infrastructure (OpenStack / Kubernetes / Ceph)
     │
     ▼
Event Bus (emit resource.created)
     │
     ▼
Subscribers (billing, audit, monitoring, notification)
```

## Data Plane

Data plane is where workloads run.

| Component | Technology |
|-----------|-----------|
| Compute Cluster | KVM hypervisor, OpenStack Nova |
| Kubernetes Cluster | Cluster API, Cilium CNI, Istio |
| Storage Cluster | Ceph (RBD, RGW, CephFS) |
| Network Fabric | OVN, Open vSwitch, VXLAN, BGP EVPN |
| DBaaS Cluster | Patroni (PG), Vitess (MySQL), Mongo Replica Set |
| Load Balancer Cluster | HAProxy, Envoy |

## Service Mapping (LotusCloud vs Hyperscalers)

| LotusCloud | AWS | GCP | Azure |
|-----------|-----|-----|-------|
| compute-service | EC2 | Compute Engine | Virtual Machines |
| container-service | ECS | Cloud Run | Container Apps |
| k8s-service | EKS | GKE | AKS |
| storage-service | S3 | Cloud Storage | Blob Storage |
| block-storage | EBS | Persistent Disk | Managed Disks |
| network-service | VPC | VPC | Virtual Network |
| sql-service | RDS | Cloud SQL | Azure SQL |
| iam-service | IAM | Cloud IAM | Entra ID |
| billing-service | Billing | Cloud Billing | Cost Management |
| monitor-service | CloudWatch | Cloud Monitoring | Azure Monitor |
| audit-service | CloudTrail | Cloud Audit | Activity Log |
| dns-service | Route 53 | Cloud DNS | Azure DNS |
| cdn-service | CloudFront | Cloud CDN | Azure CDN |
| keyvault-service | KMS | Cloud KMS | Key Vault |
| backup-service | AWS Backup | Backup/DR | Azure Backup |
| lb-service | ALB/NLB | Cloud LB | Azure LB |
| firewall-service | Network Firewall | Cloud Firewall | Azure Firewall |
| vpn-service | VPN Gateway | Cloud VPN | VPN Gateway |
| serverless-service | Lambda | Cloud Functions | Azure Functions |
| eventgrid-service | EventBridge | Eventarc | Event Grid |
| servicebus-service | SQS/SNS | Pub/Sub | Service Bus |
| policy-service | Config | Org Policy | Azure Policy |
| ddos-service | Shield | Cloud Armor | DDoS Protection |

## Multi-Region Architecture

```
LotusCloud Global
   │
   ├── Region: VN-South (Ho Chi Minh)
   │     ├── AZ: vn-south-1a
   │     ├── AZ: vn-south-1b
   │     └── AZ: vn-south-1c
   │
   ├── Region: VN-North (Hanoi)
   │     ├── AZ: vn-north-1a
   │     └── AZ: vn-north-1b
   │
   └── Region: VN-Central (Da Nang)
         └── AZ: vn-central-1a
```

Each Availability Zone contains:
- Compute Cluster
- Storage Cluster
- Network Cluster
- Control Plane Replica

## Event-Driven Architecture

LotusCloud uses event-driven architecture for async operations.

### Event Bus

Technology: Kafka (primary) or NATS (lightweight)

### Event Topics

```
resource.created
resource.deleted
resource.updated
vm.started
vm.stopped
vm.failed
cluster.created
cluster.scaled
database.created
database.backup.completed
backup.completed
billing.usage.reported
security.alert
quota.exceeded
```

### Event Flow

```
API Request
     │
     ▼
Resource Manager
     │
     ▼
Publish Event → Event Bus
                    │
                    ├── billing-service (track cost)
                    ├── audit-service (log activity)
                    ├── monitoring-service (update metrics)
                    ├── notification-service (alert user)
                    └── analytics-service (aggregate data)
```

## Cloud Resource Graph Engine

LotusCloud uses a resource graph to understand relationships between all resources.

### Graph Model

Nodes: organization, project, vpc, subnet, vm, database, cluster, bucket, user, role
Edges: belongs_to, connected_to, attached_to, member_of, has_permission

```
Organization
 └── Project
      ├── VPC
      │    ├── Subnet
      │    │    ├── VM
      │    │    │    ├── Disk (attached_to)
      │    │    │    └── NIC (connected_to)
      │    │    └── LoadBalancer
      │    └── SecurityGroup
      ├── Kubernetes Cluster
      │    └── NodePool
      └── Database Instance
           └── Backup
```

### Graph Queries

```
Find all VMs in a project
Find all resources in a VPC
Find disks attached to a VM
Find IAM permissions for a user
Find all resources using a subnet
```

## Lotus Resource Name (LRN)

Global resource identifier system (similar to AWS ARN, GCP resource name).

### Format

```
lrn:lotuscloud:{service}:{region}:{project}:{resource-type}/{resource-id}
```

### Examples

```
lrn:lotuscloud:compute:vn-south-1:proj-abc:vm/vm-001
lrn:lotuscloud:network:vn-south-1:proj-abc:vpc/vpc-101
lrn:lotuscloud:dbaas:vn-north-1:proj-abc:postgres/db-001
lrn:lotuscloud:storage:vn-south-1:proj-abc:bucket/images
lrn:lotuscloud:k8s:vn-south-1:proj-abc:cluster/k8s-prod
```

## Cloud Resource State Machine

Every resource has a lifecycle state machine.

### VM States

```
creating → starting → running → stopping → stopped → deleting → deleted
                  ↑                              │
                  └──────── restarting ←─────────┘
```

### DB Instance States

```
creating → available → modifying → available
                    → backing-up → available
                    → failing-over → available
                    → deleting → deleted
```

## Distributed Quota System

Quota hierarchy prevents resource overcommit.

```
Organization Quota
   └── Project Quota
        └── User Quota
```

### Example Project Quota

```
max_vms = 50
max_cpu = 200 cores
max_ram = 512 GB
max_storage = 10 TB
max_vpcs = 5
max_clusters = 10
max_databases = 20
```

### Enforcement Flow

```
User request (create VM)
     │
     ▼
Quota Service (check project quota)
     │
     ├── Quota available → Allow → Proceed
     └── Quota exceeded → Deny → Return 429
```

## Cloud Control Database

Central metadata database for all cloud resources.

### Schema

```
organizations (id, name, created_at)
projects (id, org_id, name, quota_id)
vpcs (id, project_id, cidr, region, zone)
subnets (id, vpc_id, cidr, zone)
vms (id, project_id, subnet_id, cpu, ram, state, zone)
clusters (id, project_id, version, node_count, state)
databases (id, project_id, engine, version, state)
volumes (id, project_id, size_gb, attached_to, state)
buckets (id, project_id, name, acl)
```

### Requirements

- High availability (Patroni cluster)
- Strong consistency
- Transactional
- Multi-zone replication

## Hyperscaler Design Principles

1. **Stateless services** — All control plane services are stateless
2. **Event-driven architecture** — Async processing via event bus
3. **Horizontal scaling** — Every service scales independently
4. **Failure isolation** — Zone, region, service, tenant isolation
5. **Eventually consistent** — Data plane operations are eventually consistent
6. **Idempotent APIs** — All mutations are idempotent
7. **Infrastructure as Code** — All provisioning via declarative config

## Technology Stack

| Layer | Technology |
|-------|-----------|
| Backend | Go, gRPC, REST, Gin |
| Event Bus | Kafka, RabbitMQ, NATS |
| Database | PostgreSQL (Patroni HA), Redis |
| IaaS | OpenStack (Nova, Neutron, Cinder, Glance, Swift, Keystone) |
| Container | Kubernetes, Cluster API, Cilium, Istio |
| Storage | Ceph (RBD, RGW, CephFS) |
| Networking | OVN, Open vSwitch, VXLAN, BGP EVPN |
| Monitoring | Prometheus, Grafana, Loki, Tempo, Jaeger |
| Security | HashiCorp Vault, mTLS, OIDC |
| IaC | Terraform, Ansible, Helm |
| Frontend | Next.js, React, Tailwind |