Open Application Model in Alibaba

Xiang Li
Alibaba Cloud
Application Management At Alibaba
Developers Operators

Serverless C
PaaS B

PaaS A Deployment
Challenges
App Instance Function
Service • Fragmentation in app layer, ~11
Job internal PaaS/Serverless
• Siloes, lack reusability and
interoperability
Route Monitor Route • Infra centric, low level primitives
HPA
Deployment Gateway Rollout leaked to end user
Gateway
Traffic Job Route Alert Traffic

Scale: Velocity:
- 10,000 nodes/cluster - 100,000 deploys/day
- 100,000 apps/cluster - 500~1000 replicas/app
- 1,000,000 containers/cluster
Infra Ops
 Container

OAM Recap
Function
What to run: Components VM
OAM Spec Cloud Resource
Operational capability：Traits

scaling
Helm, Kustomize, DSL (e.g. cdk8s) routing
manually create
ingress

packaging trigger
OAM YAML files

Helm, CNAB, OCI GitOps

OAM
deploy Implementation

Prometheus

Istio Kubernetes Cloud Resources

...
Open Application Model
Application
For example
PHP Web App
A typical OAM based app platform
How K8s capabilities are managed today?

Current state
• ~100 CRD + controllers installed in one cluster
• use annotation/webhook to discover and manage

Challenges
• Nobody knows how many capabilities the system supports
exactly, even some of them are duplicated/deprecated
• Nobody knows how many capabilities are applied to a
given application
• Nobody knows if one capability is conflict with the other
until the deployment is fail

a real world example in Alibaba PaaS

Trait Registry (WIP) 1. Register
$ kubectl oam install cronhpa-def.yaml category=scaling conflict=hpa
SUCCESS – CRD cronhpa.core.oam.dev is registered as scaling trait!

Rollout

In-place Blue-green Canary 2. Discover

$ kubectl get traits
NAME CATEGORY
blue-green rollout
Conflict
in-place rollout
hpa scaling
Scaling cronhpa scaling

HPA CronHPA
app-config.yaml

3. Auto conflict detect at deploy time

$ kubectl apply –f app-config.yaml
Conflict
ERROR – traits hpa and cronhpa MUST not be applied to same
component!
Why?
• Right level of abstraction: developer level abstraction to consolidate the application management
experience,
• Team-centric: separate concerns for developers, operators, and infra engineers.
• Platform agnostic: deploy with same YAML to serverless platform, k8s on prems, or any cloud.
• Manageability and discoverability: detect conflicted operational strategies at deploy
OAM in Alibaba at highest level

CI/CD for developers

App Marketplace
OAM Standard
Dev + Ops model
App PaaS
workload definition
Hybrid Cloud
trait definition
Users
cloud resource definition

service binding
Serverless ISV

Operator IoT
platform
App/Serverless platforms in Alibaba
• Alibaba Application Platform (EDAS)
• distributed Java application platform
• Public product
• Alibaba Operator Platform
• Stateful application platform
• Internal platform
• Alibaba Serverless Platform
• Serverless (event driven) application platform
• Both public product and internal platform
OAM + Alibaba Application PaaS (EDAS)
• Before: traditional PaaS with close ecosystem
• After: OAM as core application model, provide simplified end user experience but Balance extensibility of k8s
still keep the extensibility of k8s, make EDAS become into an open ecosystem
with best end-user experience
by OAM based abstractions
Feature OAM Category Description
• ContainerizedWorkload • component • Stateless app

• Virtual Machine • component • Legacy app

• traffic rollout • trait • Traffic shifting Reusable components and

• monitoring • trait • Prometheus configuration
traits to save engineering time
• logging • trait • SLS log collecting policy
• auto-scaler • trait • HPA
• manual-scaler • trait • Manual scaling
• rollout • trait • Blue-green deployment Discoverability for capabilities
• ingress • trait • Routing policy in the system, detect strategy
conflict at deploy

Share with other platforms

OAM + Alibaba Operator Platform
• Before: in-house app definition, lack of portability and interoperability across
clusters
• After: OAM as core application model to manage stateful applications, e.g. Kafka,
All speak OAM API, easy to
Zookeeper and Database on any cluster integrate with other OAM
systems
feature OAM Category Description
• OpenKruise • component • Alternative to StatefulSet

• Operators • component • Workloads based on operator

• In-place rollout • trait • In-place rollout OAM to decouple application

• canary • trait • Canary deployment
from runtime, easy to upgrade
infrastructure
• khpa • Trait • HPA to scale Operators

• pvc • trait • Claim persistent storage

• backup • trait • Data backup

• restore • trait • Data restore

Reusable components and
• ingress • trait • Routing policy
traits to save engineering time
• monitoring • trait • Integrated internal monitoring
policy

Share with other platforms

OAM + Alibaba Serverless Platform
Separate concerns make
developers accept serverless
• Before: in-house app definition, reinvent every wheel for serverless model way easier
• After: OAM as unified definition of various workload types, and reuse traits from may
other systems like EDAS.

Migrate from K8s/PaaS to

feature OAM Category Description
Serverless only need to update
• ContainerizedWorkload • component • Serverless container
its trait
• Function • component • FaaS workload

• Knative service (WIP) • component • Knative serving workload

• traffic-mgmt • trait • Istio based Traffic shifting

Reusable components and
• cron-scaler • trait • Cron HPA
traits to save engineering time
• manual-scaler • trait • Manual scale

• auto-scaler • trait • HPA

• sls-project, sls-logstore, sls- • component • Alibaba Cloud Resources as Connect and consume cloud
index BaaS
resources thru OAM based
BaaS
Share with other platforms
Future
PaaS A PaaS B Serverless C

Application Application Application

Configuration Configuration Configuration

Traits Components Traits Components Traits Components

Traits/Scopes Components
Service
Traffic Gateway Monitor Job Container
Binding
interpretability

Ingress Route Alert Rollout Function Virtual

Machine

Scale: Applications:
- 10,000 nodes/cluster - 100,000 deploys/day
- 100,000 apps/cluster - 500~1000 replicas/app
- 1,000,000 containers/cluster
Infra Ops
Kubernetes in Alibaba with OAM

Modular, reusable, standardized, and

manageable platform capabilities, best
engineering efficiency

Clear separate of concerns in the workflow of

app management, best communication and
collaboration efficiency • Simple, team-centric and unified experience
• Enjoy “unlimited” capabilities provided by k8s
Minimal migration effort from PaaS/K8s to OAM ecosystem - the “open” PaaS
platform

The simplest and unified approach to integrate

and consume Alibaba Cloud Resources
For end users (Dev and Ops)

For Platform
Demo - FlightTracker
Deploy and run the app on different clouds with exactly same YAML file