Freemium SaaS on K8s: Automating namespace-per-customer provisioning with GitLab CI, who's doing this?
Body:
Been running a production RKE2 cluster (3 nodes, Longhorn storage, GitLab Agent) for our main app for a while. Now we're pivoting to a freemium SaaS model and I want to sanity-check the architecture before we commit.
The Goal:
Customer signs up → Gets customername.ourapp.com → We spin up a complete isolated replica of our stack (Java backend + Postgres + ActiveMQ) in its own namespace automatically. Trial expires after 30 days → auto-cleanup.
Current Approach:
- Namespace-per-tenant (soft isolation via NetworkPolicies + ResourceQuotas)
- GitLab CI triggers the provisioning (we already use the agent for prod deploys)
- Helm templating to generate manifests per customer
- Cert-manager for subdomain TLS
- TTL controller CronJob to nuke expired trials
Each tenant gets:
- Dedicated Postgres (per-tenant PV via Longhorn, not shared DB)
- 1-2 app replicas
- 2 CPU / 4GB RAM quotas (enforced)
- Isolated ingress subdomain
The Questions:
- Scale concerns: Anyone running 100+ namespaces on a 3-node RKE2 cluster? Control plane stress or etcd size issues? We're expecting slow growth but want headroom.
- Cost efficiency: Per-tenant Postgres is "safer" but pricier than shared DB with row-level security. For freemium/trials, is the isolation worth the overhead? How do you handle the "noisy neighbor" problem without breaking the bank?
- GitLab CI vs Operator: We're using pipeline triggers right now (30-60s provisioning time). Anyone moved from CI-based provisioning to a proper Kubernetes Operator for tenant lifecycle? Worth the complexity at ~50 tenants or wait for 500?
- Subdomain routing: Using NGINX Ingress with wildcard cert. Any gotchas with custom domains later (customer wants their own domain instead of ours)?
- The "sleep" problem: For cost control, anyone implemented "sleeping" idle namespaces (scale to zero after inactivity) for free tiers? Hibernate PVs somehow?
Would love to hear war stories from anyone who's built similar "instant environment" provisioning. Especially interested in the trade-off between namespace isolation vs multi-tenancy within single deployment for B2B SaaS freemium models.
Running this on bare metal RKE2 + containerd + Longhorn if that changes anything.
- r/kubernetes
- r/devops
- r/SaaS
