API Reference on Modelplane Docs

InferenceClass

InferenceCluster

InferenceGateway

An InferenceGateway is the front door for inference requests: the only address a caller sees. It speaks the OpenAI and Anthropic APIs, authenticates callers, resolves the model a request names to a ModelService, and forwards to whichever of that service’s endpoints should serve it, translating the request for the backend that won. A Modelplane can run several, each on an InferenceCluster of its own. Run one per region to keep a caller’s traffic in its jurisdiction, or two in a region to survive losing a cluster. Modelplane runs no global load balancer: distributing callers across gateways is yours to configure, whether by geo DNS, an anycast address, or an edge of your own with these gateways as origins.

ModelCache

ModelDeployment

ModelEndpoint

A ModelEndpoint is somewhere a request can be served: one replica of a ModelDeployment, or a model at a provider like Together or Groq. It describes a backend well enough for a gateway to talk to it without knowing where it came from, which is what lets a ModelService fan over endpoints Modelplane runs and endpoints it merely buys from. Modelplane composes one per replica. You write them by hand for anything it doesn’t run.

ModelReplica

ModelService

A ModelService is one model as a caller sees it: a stable name that resolves to whichever ModelEndpoint should serve the next request. The endpoints behind it can be replicas Modelplane runs, models bought from a provider, or both, in more than one region. A caller reaches it by naming it as the model in an ordinary OpenAI or Anthropic request to any InferenceGateway that serves it. There is no per-service address.

AKSCluster

An AKSCluster provisions an Azure Kubernetes Service cluster with a dedicated resource group, virtual network, and node pools for GPU inference and system workloads. It outputs a Secret containing the cluster kubeconfig that consumers use to target the cluster. The kubeconfig embeds a client certificate, so consumers need no separate cloud identity.

EKSCluster

An EKSCluster provisions an EKS cluster with dedicated node groups for GPU inference and system workloads. It outputs a Secret containing the cluster kubeconfig that consumers use to target the cluster. The kubeconfig embeds a static bearer token that the AWS provider refreshes.

GKECluster

A GKECluster provisions a GKE cluster with dedicated node pools for GPU inference and system workloads. It outputs secrets containing the cluster kubeconfig and a GCP service account key that consumers can use to target the cluster.

NebiusCluster

A NebiusCluster provisions a Nebius Managed Service for Kubernetes (mk8s) cluster with dedicated node groups for GPU inference and system workloads. It outputs a Secret containing the cluster kubeconfig. The kubeconfig carries only the cluster endpoint and CA certificate - Nebius clusters authenticate every client through Nebius IAM - so consumers pair it with the credentials Secret of the config referenced by spec.credentials. The cluster is created in the project that config sets as its projectID; Nebius projects are bound to a region, so the project also determines where the cluster runs.

ServingStack

A ServingStack installs the serving substrate on a Kubernetes cluster: LeaderWorkerSet (Standard) or Grove and the KAI Scheduler (Dynamo), plus Gateway API, cert-manager, and Prometheus.

VultrCluster

A VultrCluster provisions a Vultr Kubernetes Engine (VKE) cluster with dedicated node pools for GPU inference and system workloads. It outputs a Secret containing the cluster kubeconfig. The kubeconfig embeds static client certificates, so consumers need nothing beyond it to reach the cluster. Note that provider-vultr only honors node pool changes at creation time; resize existing pools through VKE’s server-side autoscaler by setting maxNodeCount up front.

ModelRoute

A ModelRoute is one ModelService’s routing on one InferenceGateway: the AIGatewayRoute matching the service’s model name, plus, per endpoint, the Backend, credential and policy the gateway needs to reach it. The ModelService composes one per gateway that serves it, pinned to that gateway, and this renders onto the gateway’s cluster. It is machine-generated. kubectl get modelroutes -l modelplane.ai/service=<name> is the per-gateway view of where a service is served and whether each gateway is carrying it.