SkyPilot,
opened up.

§ 00 · PrologueWhy we cracked this open.

SkyPilot 是开源多云编排里的事实标准。表面是一行 sky launch，底下藏着一套客户机/服务器/节点守护的三段式系统、一个混合 DP + ILP 求解器、一组面向 24 个云厂商的多态抽象，以及 6562 行单文件后端。读它，是为了把"如何用一个统一控制平面调度异构基础设施"这件事彻底搞清楚 — 因为我们手头正在搭的，是它的近亲。

SkyPilot is the de facto standard for open-source multi-cloud orchestration. On the surface it is a single sky launch; underneath sits a three-tier client / server / node-daemon system, a hybrid DP + ILP solver, a set of polymorphic abstractions spanning 24 cloud vendors, and a 6,562-line single-file backend. We read it to fully understand how one unified control plane schedules heterogeneous infrastructure — because what we're building right now is a close cousin of it.

这次精读分 11 个模块（M0 → M9，加一个中途插入的 M2.5），共 6.5 小时。本档案是读后的总览：架构总图、9-stage 执行流水线、Optimizer 的 DP / ILP 路由、两层 job 状态机、加新云后端的文件改动清单 — 全部以手绘 SVG 呈现，无运行时依赖，离线可读。

This reading spans 11 modules (M0 → M9, plus an M2.5 wedged in mid-way), 6.5 hours total. This file is the post-reading overview: the architecture map, the 9-stage execution pipeline, the Optimizer's DP / ILP routing, the two-level job state machine, and the file-change checklist for adding a new cloud backend — all rendered as hand-drawn SVG, no runtime dependencies, readable offline.

— 上表为 master 分支（commit f8bb042）核心文件行数；前 4 个最大的合计已是 SkyPilot 实际"心脏"的一半以上。

— Line counts of the core files on master (commit f8bb042); the four largest alone already make up more than half of SkyPilot's actual "heart."

§ M0 · 20 minThe repository, in repose.

把 21 万行的仓库当成一个整体先看一眼。它叫 sky，主包就是 sky/，根目录还有三个有意思的伙伴：agent/（SkyPilot Agent Skills — 给 LLM agent 用的 GPU 接入封装，v0.12 新功能）、llm/（45+ 个 LLM 训练/serving 例子，从 DeepSeek-R1 到 Kimi-K2），还有 examples/amd/ — 4 个直接面向 MI 系列硬件的官方 YAML 范例。

Take in the 211K-line repo as a whole first. It is called sky; the main package is just sky/, and the root has three interesting companions: agent/ (SkyPilot Agent Skills — a GPU-access wrapper for LLM agents, new in v0.12), llm/ (45+ LLM training/serving examples, from DeepSeek-R1 to Kimi-K2), and examples/amd/ — four official YAML samples aimed directly at MI-series hardware.

公开 API 表面 · sky/__init__.py

The public API surface · sky/__init__.py

258 行的薄薄一份。它做的事情只有三件：(1) 在 import 时强制规范 HTTP/HTTPS 代理环境变量（因为 GCP SDK 对大小写敏感有 bug），(2) 重新导出 sky.client.sdk 里所有动词 API 到顶层命名空间，(3) 给 24 个云类型起别名。sky.launch 之所以不在 sky/launch.py，是因为 SkyPilot 用了门面模式 — 用户调 sky.launch() 感觉是本地函数，实际背后是 client → HTTP → server → execution pipeline 的整套链路。

A thin 258 lines. It does exactly three things: (1) at import time, force-normalizes the HTTP/HTTPS proxy environment variables (because the GCP SDK has a case-sensitivity bug), (2) re-exports all the verb APIs from sky.client.sdk into the top-level namespace, and (3) aliases the 24 cloud types. sky.launch doesn't live in sky/launch.py because SkyPilot uses the facade pattern — calling sky.launch() feels like a local function, but behind it is the full client → HTTP → server → execution pipeline chain.

# sky/__init__.py · line 83-84
# Keep this order to avoid cyclic imports
# pylint: disable=wrong-import-position
from sky import backends
from sky import batch  # noqa: F401
from sky import clouds
from sky.client.sdk import launch       # ← 重导出门面在这里
from sky.client.sdk import status
from sky.client.sdk import exec
...
AWS = clouds.AWS                          # ← 24 个云别名
GCP = clouds.GCP
Kubernetes = clouds.Kubernetes
K8s = Kubernetes                          # ← 别名的别名

那条 # Keep this order to avoid cyclic imports 注释是真实工程债 — 内部模块之间有循环依赖，导入顺序敏感。读源码时如果看到 task.py 和 resources.py 互相 import，不用惊讶。

That # Keep this order to avoid cyclic imports comment is real engineering debt — internal modules have circular dependencies, so import order is load-bearing. Don't be surprised when you see task.py and resources.py importing each other.

§ M1 · 35 minThree objects, one universe.

三个文件，三个数据类，是 SkyPilot 整个 universe 的元素周期表：

Three files, three data classes — the periodic table of SkyPilot's entire universe:

反直觉的是 — Resources 比 Task 还大。原因是 Resources 有 30+ 个 @property 暴露视图：infra / cloud / region / zone / instance_type / cpus / memory / accelerators / use_spot / disk_size / disk_tier / network_tier / image_id / ports / labels / autostop_config / priority / docker_login_config / ... 每个字段都对应一个云上的实际配置维度，24 个云全部要覆盖 — 字段就是这么多。

Counterintuitively — Resources is bigger than Task. The reason is that Resources exposes 30+ @property views: infra / cloud / region / zone / instance_type / cpus / memory / accelerators / use_spot / disk_size / disk_tier / network_tier / image_id / ports / labels / autostop_config / priority / docker_login_config / ... Each field maps to a real configuration dimension on the cloud, and all 24 clouds have to be covered — that's just how many fields there are.

YAML → Python 的翻译：_fill_in_env_vars 的 JSON 中转大法

Translating YAML → Python: the JSON round-trip trick in _fill_in_env_vars

用户写 file_mounts: { /model/llama-${SIZE}b: s3://llama-weights/${SIZE}b }，SkyPilot 怎么把 ${SIZE} 替换成实际值？直觉做法是递归遍历嵌套 dict 的每个字符串字段。SkyPilot 用了一个聪明的 trick：

The user writes file_mounts: { /model/llama-${SIZE}b: s3://llama-weights/${SIZE}b } — how does SkyPilot substitute ${SIZE} with the real value? The intuitive approach is to recursively walk every string field of the nested dict. SkyPilot uses a clever trick instead:

def _fill_in_env_vars(yaml_field, task_envs):
    yaml_field_str = json.dumps(yaml_field)           # 1) 嵌套 dict → JSON 字符串

    def replace_var(match):
        var_name = match.group(1)
        return task_envs.get(var_name, match.group(0))

    pattern = r'\$\{?\b([a-zA-Z_][a-zA-Z0-9_]*)\b\}?' # 2) regex 抓所有 $VAR / ${VAR}
    yaml_field_str = re.sub(pattern, replace_var, yaml_field_str)
    return json.loads(yaml_field_str)                 # 3) JSON 反序列化回 dict

3 行核心代码就替代了"递归遍历嵌套结构"的几十行 walker。可行性的前提：(1) schema 已经被 validate_schema 在上游校验过，(2) 不需要支持 bash 默认值语法 ${VAR:-default}（这是个挂了好几年的 TODO，line 115）。性能上每次跑一次 JSON 序列化/反序列化不是最优，但对配置加载这种"少量、一次性"场景，简洁度胜过性能。

Three lines of core code replace dozens of lines of "recursive nested-structure walker." The preconditions that make it work: (1) the schema has already been validated upstream by validate_schema, and (2) there's no need to support bash default-value syntax ${VAR:-default} (a TODO that's been open for years, line 115). Running a JSON serialize/deserialize each time isn't optimal, but for "small, one-shot" config loading, brevity wins over performance.

另外两个值得记的点

Two more things worth noting

ManagedSecretRef（line 279）— 一个 dataclass，三个字段（name / mount_path / scope_override）。它让 YAML 里能写 secrets: [secrets:HF_TOKEN, secrets:workspace:GH_PAT] 这种命名引用而不是内联值；token 实际值存在 server 端的 vault 里，YAML 自己不暴露。scope_override 决定查哪个命名空间（personal / workspace / global）。

ManagedSecretRef (line 279) — a dataclass with three fields (name / mount_path / scope_override). It lets YAML write named references like secrets: [secrets:HF_TOKEN, secrets:workspace:GH_PAT] instead of inline values; the actual token lives in the server-side vault, and the YAML never exposes it. scope_override decides which namespace to look in (personal / workspace / global).

register_task_validator（line 36）— SkyPilot 的 plugin 机制典范。一个全局列表 _task_validators，Task.validate() 时遍历调用。谁注册不显式，靠 module import 副作用触发。如果你将来想给 SkyPilot 加"AMD-specific 任务校验"（比如 run 命令里出现 cuda 关键字但 resources 是 AMD 时警告），不用 fork 主代码，写个 plugin 注册到这里即可。

register_task_validator (line 36) — a model of SkyPilot's plugin mechanism. A global list _task_validators, iterated over during Task.validate(). Who registers is implicit, triggered by module-import side effects. If you ever want to add "AMD-specific task validation" to SkyPilot (e.g. warn when the run command contains the cuda keyword but resources is AMD), you don't fork the main code — just write a plugin and register it here.

§ M2 / M2.5 · 60 minCLI · SDK · the FastAPI in the middle.

把用户敲的 sky launch foo.yaml 拆解开看，它跨越四个进程边界：终端 shell → Click CLI → Python SDK → FastAPI server → 子进程 executor。每一跳都值得展开。

Take the user's sky launch foo.yaml apart and it crosses four process boundaries: terminal shell → Click CLI → Python SDK → FastAPI server → subprocess executor. Every hop is worth unpacking.

Click CLI · sky/client/cli/command.py · 7,954 行

Click CLI · sky/client/cli/command.py · 7,954 lines

我们之前估计 2900 行，实际接近三倍 — 它把 30+ 个子命令（launch / exec / status / queue / cost_report / logs / down / stop / autostop / start / check / show-gpus / jobs / serve ...）的处理逻辑全放一起。launch 子命令处理器在 line 1235，接受 30+ 个 Click 选项 — 注意到 infra、cloud、region、zone 是并存的，老接口（cloud/region/zone）和新接口（infra）渐进迁移期共存。

We'd estimated 2,900 lines; it's nearly triple that — it lumps the handler logic for 30+ subcommands (launch / exec / status / queue / cost_report / logs / down / stop / autostop / start / check / show-gpus / jobs / serve ...) all together. The launch subcommand handler is at line 1235 and takes 30+ Click options — note that infra, cloud, region, and zone coexist: the old interface (cloud/region/zone) and the new one (infra) live side by side during a gradual migration.

task_or_dag = _make_task_or_dag_from_entrypoint_with_overrides(
    entrypoint=entrypoint,        # foo.yaml 路径
    name=name, workdir=workdir,
    cloud=cloud, region=region, zone=zone,
    gpus=gpus, cpus=cpus, memory=memory,
    instance_type=instance_type, num_nodes=num_nodes,
    use_spot=use_spot, image_id=image_id,
    env=env, secret=secret,
    ...
)
# ↑ 这一调内部走 M1 的 Task.from_yaml_config
#   合并 CLI overrides 进 envs/secrets, 跑 _fill_in_env_vars,
#   返回纯净的 Task 对象

request_id = sdk.launch(task, dryrun=dryrun, ...)
# ↑ 这一调走 HTTP，返回 request_id (不是结果！)

Python SDK · sky/client/sdk.py · 3,237 行

Python SDK · sky/client/sdk.py · 3,237 lines

关键签名揭示一切：def launch(task, ...) -> RequestId[Tuple[Optional[int], Optional[ResourceHandle]]]。返回 RequestId 而不是结果。SDK 是异步的 — 你拿到 ID 之后用 sky.stream_and_get(request_id) 才能取实际结果（或者 CLI 替你做这件事）。原因：launch 可能要跑 30 分钟（provision VM + 装依赖 + 跑 setup），HTTP 请求绝不能阻塞。

The key signature tells the whole story: def launch(task, ...) -> RequestId[Tuple[Optional[int], Optional[ResourceHandle]]]. It returns a RequestId, not a result. The SDK is async — once you have the ID, you call sky.stream_and_get(request_id) to fetch the actual result (or the CLI does it for you). The reason: a launch may run for 30 minutes (provision the VM + install dependencies + run setup), and the HTTP request must never block.

SDK 还有几个 _is_launched_by_jobs_controller / _is_launched_by_sky_serve_controller 这种内部标志 — 这暗示 Managed Jobs 和 SkyServe 内部会递归调 SDK launch 子任务（在 controller VM 上启动 worker VM）。

The SDK also carries internal flags like _is_launched_by_jobs_controller / _is_launched_by_sky_serve_controller — a hint that Managed Jobs and SkyServe recursively call SDK launch for sub-tasks internally (spinning up worker VMs from the controller VM).

FastAPI server · sky/server/server.py · 3,524 行

FastAPI server · sky/server/server.py · 3,524 lines

line 926 一行揭示骨架：

One line at line 926 reveals the skeleton:

app = fastapi.FastAPI(prefix='/api/v1', debug=True, lifespan=lifespan)

然后是 10+ 个 middleware：RBAC / RequestID / BasicAuth / BearerToken / AuthProxy / SecurityHeaders / InternalDashboardPrefix / CacheControlStatic / PathClean / GracefulShutdown / APIVersion。从 routes 看，端点包括 /token / /api/v1/auth/* / /check / /enabled_clouds 以及一长串异步任务（cleanup_upload_ids / cleanup_unreferenced_file_mounts / loop_lag_monitor 等），全部跑在 FastAPI 的 lifespan 上下文里。

Then come 10+ middlewares: RBAC / RequestID / BasicAuth / BearerToken / AuthProxy / SecurityHeaders / InternalDashboardPrefix / CacheControlStatic / PathClean / GracefulShutdown / APIVersion. From the routes, the endpoints include /token / /api/v1/auth/* / /check / /enabled_clouds plus a long list of async tasks (cleanup_upload_ids / cleanup_unreferenced_file_mounts / loop_lag_monitor and more), all running inside FastAPI's lifespan context.

Request executor · sky/server/requests/executor.py

Request executor · sky/server/requests/executor.py

这是最关键的一段，整个客户机/服务器解耦的真相在这里：

This is the most critical piece — the whole truth of the client/server decoupling lives here:

multiprocessing.set_start_method('spawn', force=True)
# On macOS, the default start method for multiprocessing is 'fork', which...

每个 launch 请求被丢进一个独立的子进程（spawn，不是 fork）跑。这才是为什么 launch 不会阻塞其他请求：它在另一个 Python 解释器里跑，FastAPI 线程立刻返回 request_id。配合 BurstableExecutor，可以根据负载动态加进程。还有一个 OnDemandThreadExecutor（line 100）专门处理"轻量同步请求在协程里的执行"。

Each launch request is dropped into its own subprocess (spawn, not fork) to run. That's exactly why launch doesn't block other requests: it runs in a separate Python interpreter, and the FastAPI thread returns the request_id immediately. Paired with BurstableExecutor, processes can be added dynamically based on load. There's also an OnDemandThreadExecutor (line 100) dedicated to "running lightweight synchronous requests inside coroutines."

§ M3 · 40 minNine stages, one pipeline.

一旦请求落到 executor 子进程，sky/execution.py（939 行）就开始按 9 个 stage 推进。我们之前 plan 时以为是 7 个 — 实际枚举有 9 个，多出来的是 CLONE_DISK（experimental，从另一个 cluster 克隆磁盘到新 cluster）和 OPTIMIZE（之前漏数）。

Once a request lands in the executor subprocess, sky/execution.py (939 lines) starts advancing through 9 stages. While planning we'd assumed 7 — the actual enum has 9, the extras being CLONE_DISK (experimental, clones a disk from another cluster into a new one) and OPTIMIZE (which we'd previously missed counting).

class Stage(enum.Enum):
    CLONE_DISK = enum.auto()
    OPTIMIZE = enum.auto()
    PROVISION = enum.auto()
    SYNC_WORKDIR = enum.auto()
    SYNC_FILE_MOUNTS = enum.auto()
    SETUP = enum.auto()
    PRE_EXEC = enum.auto()
    EXEC = enum.auto()
    DOWN = enum.auto()

每个 stage 都直接映射到 backend.method() 调用 — backend 抽象就是这套"做 9 件事"的契约：

Each stage maps directly to a backend.method() call — the backend abstraction is exactly this "do 9 things" contract:

§ M4 · 45 minThe Optimizer, DP × ILP.

Optimizer 是 SkyPilot 的技术宝石 — 1805 行做一件事：在 24 个云 × N 个 region × M 个 instance type × spot/on-demand × egress 的笛卡尔积里挑出代价最小（或时间最短）的资源分配方案。

The Optimizer is SkyPilot's technical jewel — 1,805 lines doing one thing: across the Cartesian product of 24 clouds × N regions × M instance types × spot/on-demand × egress, pick the resource allocation with the lowest cost (or shortest time).

关键的判别在第一步：拿到 Dag 之后，问它 dag.is_chain()：

The key branch happens in the first step: given the Dag, ask it dag.is_chain():

动态规划 · chain DAG 上的最短路径

Dynamic programming · shortest path on a chain DAG

状态定义：dp_best_objective[node][resources] = 用 resources 配置跑完 node 的最小累积代价。状态转移：

State definition: dp_best_objective[node][resources] = the minimum cumulative cost to finish node using the resources config. State transition:

# dp_best_objective[node][resources]
#     = my_execution_cost(node, resources)
#     + min over parent_resources of (
#           dp_best_objective[parent][parent_resources]
#         + egress_cost(parent → node, parent_resources, resources)
#       )

再加一个 dp_point_backs[node][resources] = best_parent_resources 用于反向回溯路径。这就是 Bellman 最短路径在"资源候选图"上的应用 — 每个 (node, resources) 是图上一个节点，相邻节点之间的边权是 execution_cost + egress_cost。

Add a dp_point_backs[node][resources] = best_parent_resources for backtracking the path. This is Bellman shortest-path applied to a "resource-candidate graph" — each (node, resources) is a vertex, and the edge weight between adjacent vertices is execution_cost + egress_cost.

整数线性规划 · 一般 DAG 上的双线性优化

Integer linear programming · bilinear optimization on a general DAG

ILP 公式的 docstring 是教科书级的清晰，直接抄录关键部分：

The docstring for the ILP formulation is textbook-clear; the key part is reproduced verbatim:

For cost optimization (after linearization):
  minimize_{c, e}  Σ c[v]ᵀ · k[v]     # execution costs at each node v
                 + Σ e[u,v]ᵀ · F[u,v] # egress costs on each edge (u,v)
  subject to:
    Σ c[v] == 1          for each v in V    # one-hot: pick one resource
    Σ e[u,v] == 1        for each (u,v) in E
    e[u,v] = flatten(c[u] @ c[v]ᵀ)           # linearize the bilinear term

For time (makespan) optimization:
  minimize finish_time[sink]
  subject to:
    finish_time[v] >= c[v]ᵀ·k[v] + finish_time[u] + e[u,v]ᵀ·F[u,v]
    for each (u,v) in E
    plus the same one-hot constraints

三种"特殊"路径

Three "special" paths

除了主路径，optimize_job_group（line 1037）处理 v0.12 引入的"job group"语义 — 多个 task 必须跑在同一个 cloud / region 上（典型：RL 训练里 actor 和 replay-buffer 必须在同一可用区减少延迟）。它走的是 _optimize_same_infra + _find_common_infras + _select_best_infra 这条特殊路径，把"基础设施一致"作为硬约束。

Beyond the main path, optimize_job_group (line 1037) handles the "job group" semantics introduced in v0.12 — multiple tasks that must run on the same cloud / region (typical case: in RL training the actor and replay-buffer must share an availability zone to cut latency). It takes a special path through _optimize_same_infra + _find_common_infras + _select_best_infra, treating "infrastructure consistency" as a hard constraint.

§ M5 / M6 · 75 minBackend × Cloud, the polymorphic dance.

这两层互为因果，必须一起读。backend.py（212 行）定义 Backend[ResourceHandle] 泛型抽象基类，列出 9 个 stage 对应的方法。cloud.py（1041 行）定义 Cloud 抽象基类，列出 14 个云必须实现的方法。

These two layers are mutually defining and must be read together. backend.py (212 lines) defines the generic abstract base class Backend[ResourceHandle], listing the methods for the 9 stages. cloud.py (1,041 lines) defines the abstract base class Cloud, listing the 14 methods every cloud must implement.

关键洞察是：Backend 是"调用方"，Cloud 是"被调用方"。先读 backend 你才能知道 cloud 接口为什么长那样 — 抽象方法的形状是被调用模式塑造出来的。

The key insight: Backend is the "caller," Cloud is the "callee." Read backend first and you'll understand why the cloud interface looks the way it does — the shape of the abstract methods is dictated by the call pattern.

Backend 抽象的形状

The shape of the Backend abstraction

class Backend(Generic[_ResourceHandleType]):
    def provision(self, task, to_provision_config, dryrun, ...) -> Tuple[ResourceHandle, bool]: ...
    def sync_workdir(self, handle, workdir, envs_and_secrets): ...
    def sync_file_mounts(self, handle, all_file_mounts, storage_mounts): ...
    def setup(self, handle, task, detach_setup): ...
    def execute(self, handle, task, dryrun) -> Optional[int]: ...     # → job_id
    def teardown(self, handle, terminate): ...
    def register_info(self, **kwargs): ...
    # ... 加上几个 internal _method 钩子

主实现 CloudVmRayBackend 在 cloud_vm_ray_backend.py 里，6562 行单文件。它的"主类"CloudVmRayBackend 本身从 line 3038 开始，到 line 6562 — 单个类 3500 行。文件里还有 7 个 helper 类：

The main implementation CloudVmRayBackend lives in cloud_vm_ray_backend.py, a 6,562-line single file. The "main class" CloudVmRayBackend itself starts at line 3038 and runs to line 6562 — a single class of 3,500 lines. The file also holds 7 helper classes:

名字里的"Ray"是什么 Ray

Which "Ray" is the Ray in the name

CloudVmRayBackend 里的 "Ray" 指 ray.io 项目 — 但不是用 Ray 做分布式训练，而是借用 Ray 的 cluster launcher（ray up / ray attach）来管 VM 生命周期。SkyPilot 早期是 Ray 的扩展，后来独立发展，这是历史包袱 + 设计选择。sky/skylet/ray_patches/ 子目录里能看到他们对 Ray cluster launcher 的私有 patch。

The "Ray" in CloudVmRayBackend refers to the ray.io project — but it isn't using Ray for distributed training. It borrows Ray's cluster launcher (ray up / ray attach) to manage VM lifecycles. SkyPilot started as a Ray extension and later spun off, so this is part legacy baggage, part design choice. The sky/skylet/ray_patches/ subdirectory shows their private patches to the Ray cluster launcher.

Cloud 抽象的 14 个必实现方法

The Cloud abstraction's 14 required methods

每个云（AWS / GCP / Kubernetes / SSH / Slurm / RunPod / ...）必须告诉 SkyPilot：

Every cloud (AWS / GCP / Kubernetes / SSH / Slurm / RunPod / ...) must tell SkyPilot:

regions_with_offering(instance_type, accel, use_spot, region, zone)
zones_provision_loop(...)               # 迭代 zones 用于 provision 尝试
get_zone_shell_cmd()                    # 在 VM 内拿 zone 的 shell 命令
instance_type_to_hourly_cost(it, spot)  # 给 Optimizer 喂价
accelerators_to_hourly_cost(accel)
get_egress_cost(num_gigabytes)          # 给 ILP 喂 egress 矩阵
make_deploy_resources_variables(...)    # 把 Resources 翻译成 Ray YAML 模板的填充变量
get_vcpus_mem_from_instance_type(it)
get_accelerators_from_instance_type(it)
get_default_instance_type(...)
_get_feasible_launchable_resources(r)   # 预过滤候选 — 给 Optimizer 减负
get_credential_file_mounts()
_unsupported_features_for_resources(r)
query_status(name_id_filter)            # 从云上拉 cluster 当前状态

具体实现 26 个云覆盖 — 加起来代码量惊人（aws.py 1712 行，kubernetes.py 1491 行）。这正是"加新云后端"工作的核心 — 后面 Red Line 1 会详述。

The concrete implementations cover 26 clouds — the combined code volume is staggering (aws.py is 1,712 lines, kubernetes.py 1,491). This is exactly the heart of "adding a new cloud backend" — detailed later in Red Line 1.

§ M7 · 40 minSkylet, the satellite.

最让人惊喜的发现：sky/skylet/skylet.py 只有 107 行。这个被誉为"SkyPilot 的 kubelet"的节点守护进程，本体精简到不可思议 — 它就做两件事：(1) 启动 gRPC server，(2) 跑一个 6 事件的轮询事件循环。重逻辑全在配套模块里。

The most pleasant surprise: sky/skylet/skylet.py is only 107 lines. This node daemon — dubbed "SkyPilot's kubelet" — has an absurdly lean core. It does just two things: (1) start a gRPC server, and (2) run a 6-event polling loop. All the heavy logic lives in companion modules.

EVENTS = [
    events.AutostopEvent(),                         # idle 自动关机
    events.JobSchedulerEvent(),                     # 节点本地 job 调度
    events.ManagedJobEvent(),                       # managed job 状态同步
    events.ServiceUpdateEvent(pool=False),          # serving controller 健康
    events.ServiceUpdateEvent(pool=True),           # pool 状态刷新
    events.UsageHeartbeatReportEvent(),             # 用量遥测 (10 min 一次)
]

def run_event_loop():
    for event in EVENTS:
        event.start()
    while True:
        time.sleep(events.EVENT_CHECKING_INTERVAL_SECONDS)
        for event in EVENTS:
            event.run()

gRPC server 暴露 4 个服务，全部用 protobuf 生成的 stub：

The gRPC server exposes 4 services, all using protobuf-generated stubs:

SkyletClient（cloud_vm_ray_backend.py:2843）就是这些 gRPC service 的客户端 — backend 通过它跟节点上的 skylet 通信。

SkyletClient (cloud_vm_ray_backend.py:2843) is the client for these gRPC services — the backend talks to the skylet on the node through it.

§ M8 · 35 minManaged Jobs, two state machines.

Managed Jobs 是 SkyPilot 的杀手特性 — 在 spot 实例上跑长跑作业不丢进度。它的核心设计是两层状态机，一个隔离另一个的瞬态故障：

Managed Jobs is SkyPilot's killer feature — run long jobs on spot instances without losing progress. Its core design is a two-level state machine, one isolating the transient failures of the other:

class ManagedJobStatus(enum.Enum):
    """
    The ManagedJobStatus is a higher level status than the JobStatus.
    Each managed job submitted to a cluster will have a JobStatus
    associated with it:
        JobStatus = [INIT, SETTING_UP, PENDING, RUNNING, ...]
    Whenever the cluster is preempted and recovered, the JobStatus
    transitions multiple times.

    However, a managed job only has one ManagedJobStatus on the jobs controller.
        ManagedJobStatus = [PENDING, STARTING, RUNNING, ...]
    """

翻译成人话：

In plain terms:

• JobStatus（在 worker cluster 的节点上）：被 spot 抢占时会经历 RUNNING → FAILED；恢复后新 cluster 起来又走 INIT → SETTING_UP → PENDING → RUNNING。一个 managed job 的生命周期里这个底层状态可能反复 RUNNING → FAILED → RUNNING。
• ManagedJobStatus（在独立的 jobs controller VM 上）：从用户视角看到的状态。底下被抢 N 次，这里始终是 RUNNING。只有当 controller 决定"不再 recover"了，才会进入 FAILED_NO_RESOURCE 或 FAILED_CONTROLLER。

• JobStatus (on the nodes of the worker cluster): when preempted by spot, it goes RUNNING → FAILED; after recovery a new cluster comes up and walks INIT → SETTING_UP → PENDING → RUNNING again. Over a managed job's lifetime this low-level status may go RUNNING → FAILED → RUNNING repeatedly.
• ManagedJobStatus (on the separate jobs controller VM): the status as the user sees it. No matter how many times it's preempted underneath, this stays RUNNING. Only when the controller decides "no more recovery" does it enter FAILED_NO_RESOURCE or FAILED_CONTROLLER.

这种"上层状态吸收下层抖动"的设计同样反映在 recovery_strategy.py（1024 行）里：

This "the upper status absorbs the lower jitter" design is also reflected in recovery_strategy.py (1,024 lines):

scheduler.py（466 行）单独控制"controller 进程"的并行度 — 因为每个 managed job 在 controller VM 上是一个独立 Python 进程，太多 controller 会撑爆 VM 内存。

scheduler.py (466 lines) separately controls the concurrency of the "controller processes" — because each managed job is a separate Python process on the controller VM, too many controllers will blow out the VM's memory.

§ M9 · 35 minSkyServe, scaling out.

SkyServe 是模型 serving 子系统 — 把 N 个 replica 跑在多云多区，自动扩缩，路由请求。结构上跟 Managed Jobs 镜像对称：每个 service 有独立的 controller VM，下面挂 K 个 replica VM，外加一个独立的 load balancer 进程负责把流量分发到健康的 replica。

SkyServe is the model-serving subsystem — run N replicas across multiple clouds and regions, autoscale, and route requests. Structurally it mirrors Managed Jobs: each service has its own controller VM, with K replica VMs hanging beneath it, plus a separate load balancer process that distributes traffic to the healthy replicas.

关键发现：load balancer 自己就是一个 FastAPI 进程（load_balancer.py:53: self._app = fastapi.FastAPI()），跑在 controller VM 上。意味着 user request 的 hot path 是 client → LB FastAPI → replica HTTP server，多一跳但代价可控（同 VM 内）。

Key finding: the load balancer is itself a FastAPI process (load_balancer.py:53: self._app = fastapi.FastAPI()) running on the controller VM. That means the hot path for a user request is client → LB FastAPI → replica HTTP server — one extra hop, but the cost is bounded (it's within the same VM).

spot_placer.py 是个有趣的独立模块 — 它不复用 Optimizer 的 ILP 逻辑，而是有专门策略：把 replica 分散在不同 region / zone，降低"同一时刻全军覆没"的概率。这是 serving 特有的可用性考虑。

spot_placer.py is an interesting standalone module — it does not reuse the Optimizer's ILP logic, but has a dedicated strategy: spread replicas across different regions / zones to lower the chance of "the whole fleet going down at once." This is an availability concern specific to serving.

Autoscaler 用 AutoscalerDecisionOperator 二元决策（SCALE_UP / SCALE_DOWN）+ AutoscalerDecision(operator, target) 数据类。QueueLengthAutoscaler 是一个具体实现 — 看队列深度做扩缩。这种"decision + operator"结构方便插入新策略（QPS / latency / GPU util）。

The autoscaler uses an AutoscalerDecisionOperator binary decision (SCALE_UP / SCALE_DOWN) + an AutoscalerDecision(operator, target) dataclass. QueueLengthAutoscaler is one concrete implementation — it scales based on queue depth. This "decision + operator" structure makes it easy to plug in new policies (QPS / latency / GPU util).

§ Traps · five of themWhat I wish I'd known.

新人最容易栽的五个坑 — 如果你将来读源码、做改动、或者跟 SkyPilot 团队提 PR，先记住这几条。

The five traps newcomers fall into most — if you're going to read the source, make changes, or send the SkyPilot team a PR, memorize these first.

§ Red lines · threeThe big questions.

读完整套源码，你应该能口头回答这三个大问题。每个红线问题贯穿多个模块，是真正的"懂了"指标。

After reading the whole source, you should be able to answer these three big questions out loud. Each red-line question cuts across multiple modules — the real measure of having "got it."

§ EpilogueWhat we found, and what's next.

SkyPilot 不是一个云抽象层；它是一个三段式分布式系统（client / API server / VM agent），中间用一个 9-stage 流水线把"用户意图"翻译成"VM 上跑的进程"。它的工程价值不是哪一个具体算法（虽然 Optimizer 的 DP+ILP 混合很漂亮），而是边界清晰 — Task 是请求、Resources 是规格、Backend 是执行契约、Cloud 是多态被调用方、Skylet 是节点上的事件循环、Managed Jobs 是两层状态机吸收瞬态故障、SkyServe 是多副本拓扑。每一层的职责锐利分明，加新功能（新云、新调度策略、新 serving 拓扑）有清晰的接入点。

SkyPilot is not a cloud abstraction layer; it's a three-tier distributed system (client / API server / VM agent), with a 9-stage pipeline in the middle translating "user intent" into "processes running on a VM." Its engineering value isn't any single algorithm (though the Optimizer's DP+ILP hybrid is beautiful), but its clean boundaries — Task is the request, Resources is the spec, Backend is the execution contract, Cloud is the polymorphic callee, Skylet is the node's event loop, Managed Jobs is a two-level state machine absorbing transient failures, and SkyServe is the multi-replica topology. Each layer's responsibility is sharply defined, and adding new features (a new cloud, a new scheduling policy, a new serving topology) has a clear entry point.

对你 multi-agent kernel optimization 系统的几个直接借鉴：

A few things to borrow directly for your multi-agent kernel optimization system:

• 异步 request_id 模式 — agent 任务投递返回 ID，长跑用 stream/poll 拿结果。直接照搬 sky/server/requests/executor.py 的 spawn 多进程模式。
• JSON 中转字符串替换 — agent 配置里的 ${MI300X_HOSTS} 模板替换，三行代码搞定，参考 M1 的 _fill_in_env_vars。
• 两层状态机吸收抖动 — kernel benchmark 偶发 OOM / driver hang 应该被下层状态机吸收，agent 任务的上层状态保持稳定。参考 M8 的 JobStatus vs ManagedJobStatus。
• "daemon 是 wiring, logic 在 lib" — agent worker 本体保持 100 行级别，把 benchmark / profile / analysis 拆成独立 lib。参考 M7 的 skylet 107 行。
• 插件化 validator / strategy — recovery strategy 可插拔，task validator 可注册。给你的 kernel optimization 加新策略时不要 fork 主代码，靠 plugin 注册。参考 register_task_validator 和 recovery_strategy.py。

• The async request_id pattern — agent task submission returns an ID; long runs use stream/poll to fetch results. Copy the spawn-based multiprocess pattern of sky/server/requests/executor.py outright.
• JSON round-trip string substitution — template substitution like ${MI300X_HOSTS} in agent configs, done in three lines; see _fill_in_env_vars from M1.
• A two-level state machine absorbs jitter — intermittent OOM / driver hang during a kernel benchmark should be absorbed by the lower state machine while the agent task's upper status stays stable. See JobStatus vs ManagedJobStatus from M8.
• "The daemon is wiring, the logic is in libs" — keep the agent worker proper at the 100-line scale and split benchmark / profile / analysis into standalone libs. See skylet's 107 lines from M7.
• Pluggable validators / strategies — the recovery strategy is pluggable, the task validator is registerable. When adding a new strategy to your kernel optimization, don't fork the main code — register a plugin. See register_task_validator and recovery_strategy.py.

— Fin.

— Fin.