ModelRouter

inference.llmkube.dev / v1alpha1

apiVersion: inference.llmkube.dev/v1alpha1 kind: ModelRouter metadata: name: example

apiVersion string

APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

kind string

Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

metadata object

spec object required

spec defines the desired state of ModelRouter

agentGatewayRef object

AgentGatewayRef identifies the pre-installed agentgateway data plane the generated InferencePool/HTTPRoute/model-extraction AgentgatewayPolicy attach to when DataPlane is "AgentGateway". Required in AgentGateway mode; ignored otherwise. The Gateway, the Gateway API Inference Extension CRDs (InferencePool at inference.networking.k8s.io/v1), the agentgateway.dev CRDs, and an Endpoint Picker (EPP) service are documented prerequisites; LLMKube does not install or own them. The agentgateway controller must run with Inference-Pool backendRef support (inferenceExtension.enabled=true in the agentgateway chart). The EndpointPicker is referenced by every generated InferencePool so agentgateway can select a model-server endpoint per request. The Gateway may be in another namespace: the generated AgentgatewayPolicy then lives in the Gateway's namespace (its targetRefs are same-namespace), is not owner-referenced, and is cleaned up via a finalizer on this ModelRouter. Because that policy targets the whole Gateway, its body-model extraction runs on EVERY request through that Gateway, including routes LLMKube did not generate. The Gateway must not already carry another PreRouting transformation policy: the router is refused (ConflictingPreRoutingPolicy) rather than silently shadowed. agentgateway keeps the OLDEST same-specificity policy's transformation, so LLMKube does not retract its policy on conflict; the condition clears when the foreign policy is removed, and the reconciler rechecks every two minutes (a foreign policy created after the router went Ready is otherwise silently shadowed). agentgateway inspects at most 2 MiB of the request body by default (its DEFAULT_BUFFER_LIMIT); larger bodies (long contexts, images) are not routed on this path - the CEL extraction fails and the model header is removed. A Gateway owner can raise the limit with a frontend.http.maxBufferSize policy, a different field that does not conflict with the extraction policy.

endpointPicker string

EndpointPicker names the router-wide default EPP Service. A backend's own endpointPicker overrides it. A Gateway API Inference Extension EPP binds to exactly one InferencePool, so the default serves a router whose backends all reference one InferenceService; with several, set spec.backends[].endpointPicker. The Service lives in this ModelRouter's namespace: GIE v1's endpointPickerRef has no namespace field.

endpointPickerPort integer

EndpointPickerPort is the port the default EPP Service listens on. Set together with endpointPicker.

format: int32

minimum: 1

maximum: 65535

hostnames []string

Hostnames limit the generated HTTPRoute to these hosts. routing: Model only, where it or sectionName is required.

minItems: 1

maxItems: 16

name string required

Name is the agentgateway Gateway's name.

namespace string

Namespace is the Gateway's namespace. Empty means the ModelRouter's own namespace. A different namespace is supported: the generated model-extraction AgentgatewayPolicy then lives in the Gateway's namespace (it cannot be owner-referenced across namespaces and is cleaned up via a finalizer on this ModelRouter), while the pools and route stay here. The pools, the referenced InferenceServices, and the EndpointPicker Service all live in this ModelRouter's namespace regardless.

routing string

Routing selects the compile target. "InferencePool" (the default) is the Gateway API Inference Extension path described on spec.agentGatewayRef. "Model" uses agentgateway's model router (v1.6.0+): one AgentgatewayModel per backend, matched on its displayName (else name), behind one catch-all HTTPRoute. It needs no Inference Extension CRDs or Endpoint Picker. spec.rules, spec.defaultRoute, external backends, resolution: endpoint, and per-backend endpoint pickers are refused in this mode.

enum: InferencePool, Model

sectionName string

SectionName names the Gateway listener the generated HTTPRoute attaches to. routing: Model only, where it or hostnames is required, because the route takes every path on its listeners.

pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$

minLength: 1

maxLength: 253

backends []object required

Backends are the candidate destinations the router can dispatch to. Order is not significant; selection is rule-driven. At least one backend must be declared.

minItems: 1

balancing object

Balancing places requests across this backend's pods by load and keeps a session on its pod until it runs hot. Requires resolution: endpoint. Unset keeps round-robin. Applies to the Proxy data plane only; ignored by the Gateway and AgentGateway data planes.

mode string

Mode is RoundRobin (today's behavior) or LoadAware: the pod with the least work, read from the engine's own metrics where LLMKube has a profile for the runtime (vLLM) and from in-flight requests otherwise.

enum: RoundRobin, LoadAware

prefixAffinity object

PrefixAffinity sends a request to the pod most likely to hold its prompt's prefix in KV cache, traded against that pod's load. The router remembers which prompt blocks it sent to each pod, so it needs no session header and no engine support. Set it (an empty object takes the defaults) to turn it on. It requires mode: LoadAware; with RoundRobin it is ignored and the router emits a PrefixAffinityIgnored warning event.

blockBytes integer

BlockBytes is the size of the prompt blocks the router hashes, in bytes of the request's tools, system prompt and messages (about four bytes per token). A shared prefix counts in whole blocks, so smaller blocks catch shorter shared prefixes and cost more memory. Defaults to 1024.

format: int32

minimum: 256

maximum: 16384

maxBlocksPerEndpoint integer

MaxBlocksPerEndpoint bounds the blocks the router remembers per pod; the least recently sent go first. The default, 16384, covers 16 MiB of prompt (about 4M tokens) per pod at the default block size, more than a pod's KV cache holds.

format: int32

minimum: 1024

maximum: 262144

sessionAffinity object

SessionAffinity places a new session on the least-loaded pod. The session stays there unless that pod is queuing and clearly busier than another pod, and then it stays where it moved. It requires mode: LoadAware; with RoundRobin it is ignored and the router emits a SessionAffinityIgnored warning event.

headers []string required

Headers identify a session, in priority order; the first present with a non-empty value is the key. Compared case-insensitively. Only a hash of the value is used. Each name is an RFC 7230 header token.

minItems: 1

maxItems: 8

idleTimeout string

IdleTimeout is how long a session may go without a request before the router forgets which pod it was on. The next request from that session is placed again by load. Defaults to 30m when unset. Must be between 1m and 24h.

maxLoadPercent integer

MaxLoadPercent is deprecated and has no effect since this release: sessions are now placed from the session binding table, not spilled by a load threshold. It is still accepted so existing manifests keep applying, and will be removed in a later API version.

format: int32

minimum: 101

maximum: 1000

capabilities []string

Capabilities advertised by this backend. Rules can require capabilities (e.g. ["tools", "vision", "long-context"]) to filter candidates.

costPerMillionTokens object

CostPerMillionTokens is informational. Used for cost-aware routing metrics and audit-log enrichment. Values are USD.

completionUSD string

CompletionUSD is the cost per million completion (output) tokens, in USD.

pattern: ^[0-9]+(\.[0-9]+)?$

promptUSD string

PromptUSD is the cost per million prompt (input) tokens, in USD.

pattern: ^[0-9]+(\.[0-9]+)?$

displayName string

DisplayName is an optional freeform label published as the model id on /v1/models and used by BackendNameMatch to resolve a request's model field to a backend. When unset, Name is used for both purposes (current behavior). This lets the k8s-safe Name differ from the user-facing model identifier (e.g. Name "claude-opus-4" with DisplayName "claude-opus-4-20250514").

endpointPicker object

EndpointPicker selects the Endpoint Picker (EPP) that chooses a pod for this backend's InferencePool on dataPlane: AgentGateway. When unset, the backend uses the picker another backend on the same InferenceService names, else the router-wide spec.agentGatewayRef.endpointPicker. A Gateway API Inference Extension EPP binds to exactly one InferencePool, so a router with backends on several InferenceServices sets a picker per backend. Ignored by the Proxy and Gateway data planes.

service object

Service names an EPP Service in the ModelRouter's namespace. GIE v1's endpointPickerRef has no namespace field, so the pool references it same-namespace.

name string required

Name is the EPP Service's name.

minLength: 1

port integer required

Port is the port the EPP Service listens on.

format: int32

minimum: 1

maximum: 65535

external object

External describes an out-of-cluster provider (Anthropic, OpenAI, or a LiteLLM proxy). Mutually exclusive with InferenceServiceRef.

credentialsSecretRef object

CredentialsSecretRef points to a Kubernetes Secret containing the provider credentials. Conventional keys: ANTHROPIC_API_KEY, OPENAI_API_KEY, LITELLM_MASTER_KEY. The router-proxy reads these as environment variables.

name string

Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names

model string required

Model is the upstream model identifier passed through to the provider (e.g. "claude-opus-4-7", "gpt-5", a LiteLLM model alias).

provider string required

Provider identifies the upstream API surface. For "litellm", URL must point at a running LiteLLM proxy speaking OpenAI-compatible API. For first-party providers, URL is optional (provider defaults apply).

enum: anthropic, openai, bedrock, vertex_ai, litellm

url string

URL is the base URL for the provider. Required for "litellm"; optional for first-party providers, which use their published default.

healthCheck object

HealthCheck overrides the default health probe applied to this backend by the router-proxy.

intervalSeconds integer

IntervalSeconds is how often the router-proxy probes the backend.

format: int32

minimum: 1

path string

Path is the HTTP path probed for health. Defaults to "/health" for local backends and to the provider's documented health route for external providers.

timeoutSeconds integer

TimeoutSeconds is the maximum time a single probe may take.

format: int32

minimum: 1

inferenceServiceRef object

InferenceServiceRef references an in-cluster InferenceService. Mutually exclusive with External.

name string

modelInfo object

ModelInfo describes what this backend's model accepts and how much context it holds. The proxy publishes it on the backend's /v1/models entry, where clients decide whether to send images and how to size their context. Informational only: the proxy does not enforce it. Unset, the entry carries only the model id. Applies to the Proxy data plane only; the Gateway and AgentGateway data planes ignore it.

contextLength integer

ContextLength is the context window one request gets, in tokens. When unset, the controller fills it from the InferenceService: vllmConfig.maxModelLen (or --max-model-len in extraArgs) for vLLM, and contextSize divided across parallelSlots for llama.cpp. Other runtimes, and llama.cpp extraArgs that set those flags, publish none.

format: int64

minimum: 1

inputModalities []string

InputModalities the model accepts. When unset, a backend whose capabilities include "vision" publishes ["text", "image"], and any other backend ["text"]. When set, it wins over capabilities.

maxItems: 5

outputModalities []string

OutputModalities the model produces. Defaults to ["text"].

maxItems: 5

name string required

Name is the stable identifier used by rules and observability labels. Must be lowercase alphanumeric or '-'.

pattern: ^[a-z0-9][a-z0-9-]{0,62}$

resolution string

Resolution selects how a local (inferenceServiceRef) backend's upstream address is resolved. "service" (the default) dispatches to the InferenceService's ClusterIP Service DNS name, so a keep-alive client can pin to one pod through conntrack. "endpoint" has the router-proxy track the ready pods behind the Service itself and balance requests across them. External backends ignore this field.

enum: service, endpoint

tier string

Tier classifies the backend for rule matching. "local" backends are served from inside the cluster; "cloud" backends egress the cluster boundary. Fail-closed rules can only route to local-tier backends.

enum: local, cloud

timeout string

Timeout caps how long the proxy waits for this backend to begin sending response headers. When set it overrides the proxy default for dispatches that target this backend. Resolution order at dispatch time: rule.timeout || backend.timeout || proxy default (ModelRouter.spec.proxy.responseHeaderTimeout). Useful when backends in the same router have wildly different P99 envelopes (in-cluster vLLM vs Anthropic global LB).

weight integer

Weight is used for the "weighted" routing strategy. Higher values receive proportionally more traffic. Ignored for other strategies. Default 1 when unset. 0 drains the backend from weighted rules: it receives no traffic from them, even while every other backend in the rule is unavailable (the Gateway API meaning of a zero weight, which both data planes follow). A weighted rule needs at least one backend with a positive weight.

format: int32

minimum: 0

dataPlane string

DataPlane selects how this ModelRouter serves traffic. "Proxy" (default) provisions the managed router-proxy Deployment + Service and routes in-process per the rules below (today's behavior, fully back-compat). "Gateway" compiles the backends and rules onto a pre-installed Envoy AI Gateway: a Backend + AIServiceBackend per InferenceServiceRef backend, a multi-rule AIGatewayRoute, and a retry/failover BackendTrafficPolicy. In Gateway mode the router-proxy is NOT provisioned. Requires the aigw CRDs to be installed; when they are absent the gateway resources are not generated and a condition explains why. "AgentGateway" compiles the backends and rules onto a pre-installed agentgateway data plane: one AgentgatewayBackend (custom provider targeting an InferencePool) per InferenceServiceRef backend and a multi-rule HTTPRoute matching on the X-Gateway-Base-Model-Name header. In AgentGateway mode the router-proxy is NOT provisioned. Requires the agentgateway.dev CRDs and a pre-installed agentgateway Gateway; when they are absent the resources are not generated and a condition explains why.

enum: Proxy, Gateway, AgentGateway

defaultRoute string

DefaultRoute names a backend used when no rule matches. Must reference the Name of an entry in Backends.

defaultRouteStrategy string

DefaultRouteStrategy decides what happens when no rule matches. "Static" (default) routes to DefaultRoute. "BackendNameMatch" first tries to match the request's model to a backend Name, falling back to DefaultRoute only if none matches. BackendNameMatch makes every backend, including cloud-tier ones, directly addressable by name; sensitive-data protection still relies on a matching failClosed rule, which is evaluated first and therefore continues to gate before any name match.

enum: Static, BackendNameMatch

endpoint object

Endpoint defines the Kubernetes Service the router-proxy is exposed through. Mirrors the shape used by InferenceService.

apiKeySecretRef object

APIKeySecretRef names a Secret key holding the API key callers must present as "Authorization: Bearer <key>". vLLM reads it as VLLM_API_KEY; SGLang's launcher writes LLMKUBE_API_KEY to a private, memory-backed config file and unsets the variable before starting SGLang. kubectl exec sessions still receive LLMKUBE_API_KEY from the container's Secret reference. The key stays out of the pod spec and OS process arguments, but SGLang may still print api_key in its startup log (verified on v0.5.19). Treat pod logs and any downstream log storage as credentials until upstream redacts api_key and admin_api_key. vLLM does not have this command-line/log exposure. Both runtimes leave /health and /metrics open, so probes keep working. Supported for runtime vllm and sglang. A key change takes effect when the pods restart. An --api-key already in extraArgs wins; LLMKube then adds nothing. Cannot be combined with gateway.enabled: the gateway does not forward the key. Not supported on ModelRouter.

key string required

The key of the secret to select from. Must be a valid secret key.

name string

optional boolean

Specify whether the Secret or its key must be defined

externalTrafficPolicy string

ExternalTrafficPolicy sets the Service's externalTrafficPolicy. Local preserves the client source IP on a NodePort or LoadBalancer, which an IP allowlist (NetworkPolicy) needs. Unset keeps the Kubernetes default (Cluster). Requires endpoint.type NodePort or LoadBalancer.

enum: Cluster, Local

gateway object

Gateway opts this InferenceService into Envoy AI Gateway exposure. When set and Enabled, the operator generates the Backend / AIServiceBackend / AIGatewayRoute resources that front this service through a pre-installed Envoy AI Gateway. nil (the default) preserves today's behavior (no gateway resources). The Envoy AI Gateway stack and the referenced Gateway are a documented prerequisite; LLMKube does not install or own them.

enabled boolean

Enabled is the opt-in switch. When false (or when Gateway is nil), the operator generates no gateway resources for this InferenceService.

gatewayRef object required

GatewayRef identifies the pre-installed Gateway (gateway.networking.k8s.io) the generated AIGatewayRoute attaches to. The Gateway typically lives in a dedicated gateway namespace; cross-namespace attachment requires the Gateway listener's allowedRoutes.namespaces to permit this InferenceService's namespace (a documented prerequisite for the MVP; the operator does not generate ReferenceGrants or touch the listener).

name string required

Name is the Gateway's name.

namespace string

Namespace is the Gateway's namespace. Empty means the InferenceService's own namespace.

modelName string

ModelName is the OpenAI "model" string clients send, matched by the generated route rule (the x-ai-eg-model header the gateway's ext_proc populates from the request body). Defaults to ModelRef, falling back to the InferenceService name when ModelRef is empty.

nodePort integer

NodePort is the specific NodePort to pin when endpoint.type is NodePort. If set, the Service will use this exact port instead of auto-assigning from the 30000-32767 range. This provides a stable external endpoint across redeployments.

format: int32

minimum: 30000

maximum: 32767

path string

Path is the HTTP path for the inference endpoint

port integer

Port is the service port

format: int32

minimum: 1

maximum: 65535

type string

Type is the Kubernetes service type (ClusterIP, NodePort, LoadBalancer)

enum: ClusterIP, NodePort, LoadBalancer

gatewayRef object

GatewayRef identifies the pre-installed Gateway (gateway.networking.k8s.io) the generated AIGatewayRoute attaches to when DataPlane is "Gateway". Required in Gateway mode; ignored in Proxy mode. The Gateway and the Envoy AI Gateway stack are a documented prerequisite; LLMKube does not install or own them. Cross-namespace attachment requires the Gateway listener's allowedRoutes.namespaces to permit this ModelRouter's namespace.

name string required

Name is the Gateway's name.

namespace string

Namespace is the Gateway's namespace. Empty means the InferenceService's own namespace.

mcpServer object

MCPServer optionally exposes this router as a Model Context Protocol endpoint. Inactive until the Phase 3 MCP feature lands; the field is reserved in the schema for forward compatibility.

enabled boolean

Enabled toggles MCP exposure. Default false. When true (after Phase 3 lands), the router-proxy serves an MCP endpoint at /mcp using Streamable HTTP transport and OAuth 2.1.

policy object

Policy holds cross-cutting controls (budgets, classification, audit).

auditLog object

AuditLog controls structured audit emission. Auditing is always on; this field tunes the destination and verbosity.

callerHeader string

CallerHeader names a request header whose value is added to every audit entry as caller, for example X-LLMKube-Client set by each client. The client chooses the value, so it is a label rather than an identity. It is cut to 128 bytes, and characters outside printable ASCII are replaced with '?'. A header that carries a credential (Authorization, Proxy-Authorization, Cookie, X-Api-Key, Api-Key, X-Goog-Api-Key) is refused.

pattern: ^[A-Za-z0-9!#$%&'*+.^_|~-]+$

maxLength: 64

filePath string

FilePath is the destination when Sink=file. Must be writable inside the router-proxy container. Defaults to "/var/log/mlx-router/audit.log".

includeRequestBody boolean

IncludeRequestBody, when true, includes the OpenAI request body in every audit entry. Disabled by default for size and privacy.

logClientAddress boolean

LogClientAddress, when true, adds the address of the request's sender to every audit entry as client. It is the TCP peer: in-cluster callers keep their pod IP through the ClusterIP. When the peer is in TrustedProxyCIDRs, it is the first address in X-Forwarded-For, read from the right, that is not a trusted proxy. Off by default, because a client address is personal data in some deployments.

sink string

Sink selects the audit-log destination. "stdout" (default) emits one JSON object per line to the proxy container stdout, where it can be collected by the cluster log stack. "file" writes to FilePath inside the proxy container. "otlp" forwards entries to an OTel collector as log records.

enum: stdout, file, otlp

trustedProxyCIDRs []string

TrustedProxyCIDRs lists the proxies in front of the router (an ingress, a gateway) whose X-Forwarded-For is believed, as CIDRs or single addresses. Any client can send X-Forwarded-For, so with this unset the header is ignored and client is always the TCP peer. Requires logClientAddress. The controller validates the entries.

maxItems: 32

auth object

Auth configures request authentication. In dataPlane: Gateway mode it compiles to an Envoy AI Gateway SecurityPolicy that validates inbound JWTs and maps a verified claim onto a trusted header before any model dispatch. nil means no authentication is enforced. Authentication only; per-team model allowlists (authorization) are a separate surface.

allowlists []object

Allowlists restricts which verified team may reach which model (authorization), as a sibling of JWT (authentication). Each entry grants a team the models it may reach. JWT proves identity; Allowlists decide what that identity is permitted to do. Empty or nil means NO authorization is enforced: any authenticated request reaches any model (the authentication-only behavior of slice 2d-core, so adding this field cannot retroactively lock out an existing router). A non-empty Allowlists flips the generated SecurityPolicy to default-Deny: only the named teams (and, per entry, only their listed models) are allowed, and every other verified team is rejected with HTTP 403. Authorization requires authentication: Allowlists set without JWT is rejected fail-loud (you cannot authorize on an unverified claim), as is an entry with an empty Team or a duplicate Team. In dataPlane: Gateway mode these compile to the authorization block of the SAME SecurityPolicy JWT generates: one Allow rule per entry whose principal matches the verified TeamClaim value (and, when the entry lists models, the resolved x-ai-eg-model header).

models []string

Models is the set of model names this team may reach. Empty means the team may reach all models (identity-only allow). Each value matches the resolved model name (the x-ai-eg-model header), the same value spec.rules[].match.models route on.

team string required

Team is the verified teamClaim value this entry grants access to.

minLength: 1

jwt object

JWT enables JSON Web Token validation. When set (in dataPlane: Gateway mode) the gateway rejects requests without a valid token with HTTP 401 before any model dispatch, and maps the configured claim onto a trusted header.

headerKey string

HeaderKey is the request header the verified TeamClaim value lands in. Downstream team-scoped budgets key on this header. Defaults to "x-llmkube-team", matching the budget default.

issuer string required

Issuer is the OIDC issuer URL that must match the token's "iss" claim.

minLength: 1

jwksURI string required

JWKSURI is the remote JWKS endpoint the gateway fetches signing keys from to verify token signatures.

minLength: 1

provider string required

Provider is a short name for the JWT provider (e.g. "keycloak"). It labels the provider in the generated SecurityPolicy.

minLength: 1

teamClaim string required

TeamClaim is the JWT claim that identifies the tenant (e.g. "team"). Its verified value is copied into HeaderKey.

minLength: 1

budgets []object

Budgets caps token and dollar consumption per scope over a rolling window. Empty list means no budget enforcement.

headerKey string

HeaderKey is the request header carrying the team identifier when Scope=team. Defaults to "x-llmkube-team".

maxTokens integer

MaxTokens caps total tokens (prompt + completion) over the window. Either MaxTokens or MaxUSD (or both) must be set.

format: int64

minimum: 1

maxUSD string

MaxUSD caps total estimated cost in USD over the window. Cost is computed from RouterBackend.CostPerMillionTokens.

pattern: ^[0-9]+(\.[0-9]+)?$

name string required

Name identifies this budget for metrics, status, and audit logs.

pattern: ^[a-z0-9][a-z0-9-]{0,62}$

ruleName string

RuleName is required when Scope=rule. References a RouterRule.Name.

scope string required

Scope determines what the budget applies to. "router" caps all traffic through this ModelRouter. "rule" caps traffic matching a single named rule (see RuleName). "team" caps traffic identified by a request header (see HeaderKey).

enum: router, rule, team

windowSeconds integer

WindowSeconds is the rolling window over which the cap is evaluated.

format: int32

minimum: 1

classification object

Classification configures how the router determines the data classification of an inbound request.

headerKey string

HeaderKey is the request header carrying the classification. Defaults to "x-llmkube-classification".

mode string

Mode determines how the router determines a request's classification. "header-only" (default) trusts the request header (HeaderKey, defaults to x-llmkube-classification) and never reads a classification from the request body. It is the only implemented mode. "detector" and "hybrid" are reserved for a future in-proxy content classifier (#441) and are refused today. On dataPlane: Proxy the webhook rejects them and a router that sets either is not Validated (reason ClassifierNotImplemented when nothing else in the spec is invalid). On dataPlane: Gateway and AgentGateway a dataClassification match under either is refused.

enum: header-only, detector, hybrid

sensitiveClassifications []string

SensitiveClassifications are the classification values that trigger fail-closed validation: any rule matching one of these values must have FailClosed=true and reference only local-tier backends. Defaults to ["pii", "phi"].

proxy object

Proxy configures the managed router-proxy Deployment (replicas, image override for air-gapped sites, resources) and the name of its Service. Sensible defaults apply when omitted.

affinity object

Affinity is passed through to the router-proxy pod, for node affinity or pod anti-affinity against the proxy's own replicas. The proxy pods carry the stable labels app: <router>-router-proxy and inference.llmkube.dev/model-router: <router> to select on.

nodeAffinity object

Describes node affinity scheduling rules for the pod.

preferredDuringSchedulingIgnoredDuringExecution []object

The scheduler will prefer to schedule pods to nodes that satisfy the affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. The node that is most preferred is the one with the greatest sum of weights, i.e. for each node that meets all of the scheduling requirements (resource request, requiredDuringScheduling affinity expressions, etc.), compute a sum by iterating through the elements of this field and adding "weight" to the sum if the node matches the corresponding matchExpressions; the node(s) with the highest sum are the most preferred.

preference object required

A node selector term, associated with the corresponding weight.

matchExpressions []object

A list of node selector requirements by node's labels.

key string required

The label key that the selector applies to.

operator string required

Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.

values []string

An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch.

matchFields []object

A list of node selector requirements by node's fields.

key string required

The label key that the selector applies to.

operator string required

Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.

values []string

weight integer required

Weight associated with matching the corresponding nodeSelectorTerm, in the range 1-100.

format: int32

requiredDuringSchedulingIgnoredDuringExecution object

nodeSelectorTerms []object required

Required. A list of node selector terms. The terms are ORed.

matchExpressions []object

A list of node selector requirements by node's labels.

key string required

The label key that the selector applies to.

operator string required

Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.

values []string

matchFields []object

A list of node selector requirements by node's fields.

key string required

The label key that the selector applies to.

operator string required

Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.

values []string

podAffinity object

Describes pod affinity scheduling rules (e.g. co-locate this pod in the same node, zone, etc. as some other pod(s)).

preferredDuringSchedulingIgnoredDuringExecution []object

The scheduler will prefer to schedule pods to nodes that satisfy the affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. The node that is most preferred is the one with the greatest sum of weights, i.e. for each node that meets all of the scheduling requirements (resource request, requiredDuringScheduling affinity expressions, etc.), compute a sum by iterating through the elements of this field and adding "weight" to the sum if the node has pods which matches the corresponding podAffinityTerm; the node(s) with the highest sum are the most preferred.

podAffinityTerm object required

Required. A pod affinity term, associated with the corresponding weight.

labelSelector object

A label query over a set of resources, in this case pods. If it's null, this PodAffinityTerm matches with no Pods.

matchExpressions []object

matchExpressions is a list of label selector requirements. The requirements are ANDed.

key string required

key is the label key that the selector applies to.

operator string required

operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.

values []string

values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.

matchLabels object

matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.

matchLabelKeys []string

MatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both matchLabelKeys and labelSelector. Also, matchLabelKeys cannot be set when labelSelector isn't set.

mismatchLabelKeys []string

MismatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both mismatchLabelKeys and labelSelector. Also, mismatchLabelKeys cannot be set when labelSelector isn't set.

namespaceSelector object

A label query over the set of namespaces that the term applies to. The term is applied to the union of the namespaces selected by this field and the ones listed in the namespaces field. null selector and null or empty namespaces list means "this pod's namespace". An empty selector ({}) matches all namespaces.

matchExpressions []object

matchExpressions is a list of label selector requirements. The requirements are ANDed.

key string required

key is the label key that the selector applies to.

operator string required

operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.

values []string

matchLabels object

namespaces []string

namespaces specifies a static list of namespace names that the term applies to. The term is applied to the union of the namespaces listed in this field and the ones selected by namespaceSelector. null or empty namespaces list and null namespaceSelector means "this pod's namespace".

topologyKey string required

This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching the labelSelector in the specified namespaces, where co-located is defined as running on a node whose value of the label with key topologyKey matches that of any node on which any of the selected pods is running. Empty topologyKey is not allowed.

weight integer required

weight associated with matching the corresponding podAffinityTerm, in the range 1-100.

format: int32

requiredDuringSchedulingIgnoredDuringExecution []object

If the affinity requirements specified by this field are not met at scheduling time, the pod will not be scheduled onto the node. If the affinity requirements specified by this field cease to be met at some point during pod execution (e.g. due to a pod label update), the system may or may not try to eventually evict the pod from its node. When there are multiple elements, the lists of nodes corresponding to each podAffinityTerm are intersected, i.e. all terms must be satisfied.

labelSelector object

A label query over a set of resources, in this case pods. If it's null, this PodAffinityTerm matches with no Pods.

matchExpressions []object

matchExpressions is a list of label selector requirements. The requirements are ANDed.

key string required

key is the label key that the selector applies to.

operator string required

operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.

values []string

matchLabels object

matchLabelKeys []string

mismatchLabelKeys []string

namespaceSelector object

matchExpressions []object

matchExpressions is a list of label selector requirements. The requirements are ANDed.

key string required

key is the label key that the selector applies to.

operator string required

operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.

values []string

matchLabels object

namespaces []string

topologyKey string required

podAntiAffinity object

Describes pod anti-affinity scheduling rules (e.g. avoid putting this pod in the same node, zone, etc. as some other pod(s)).

preferredDuringSchedulingIgnoredDuringExecution []object

The scheduler will prefer to schedule pods to nodes that satisfy the anti-affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. The node that is most preferred is the one with the greatest sum of weights, i.e. for each node that meets all of the scheduling requirements (resource request, requiredDuringScheduling anti-affinity expressions, etc.), compute a sum by iterating through the elements of this field and subtracting "weight" from the sum if the node has pods which matches the corresponding podAffinityTerm; the node(s) with the highest sum are the most preferred.

podAffinityTerm object required

Required. A pod affinity term, associated with the corresponding weight.

labelSelector object

A label query over a set of resources, in this case pods. If it's null, this PodAffinityTerm matches with no Pods.

matchExpressions []object

matchExpressions is a list of label selector requirements. The requirements are ANDed.

key string required

key is the label key that the selector applies to.

operator string required

operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.

values []string

matchLabels object

matchLabelKeys []string

mismatchLabelKeys []string

namespaceSelector object

matchExpressions []object

matchExpressions is a list of label selector requirements. The requirements are ANDed.

key string required

key is the label key that the selector applies to.

operator string required

operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.

values []string

matchLabels object

namespaces []string

topologyKey string required

weight integer required

weight associated with matching the corresponding podAffinityTerm, in the range 1-100.

format: int32

requiredDuringSchedulingIgnoredDuringExecution []object

If the anti-affinity requirements specified by this field are not met at scheduling time, the pod will not be scheduled onto the node. If the anti-affinity requirements specified by this field cease to be met at some point during pod execution (e.g. due to a pod label update), the system may or may not try to eventually evict the pod from its node. When there are multiple elements, the lists of nodes corresponding to each podAffinityTerm are intersected, i.e. all terms must be satisfied.

labelSelector object

A label query over a set of resources, in this case pods. If it's null, this PodAffinityTerm matches with no Pods.

matchExpressions []object

matchExpressions is a list of label selector requirements. The requirements are ANDed.

key string required

key is the label key that the selector applies to.

operator string required

operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.

values []string

matchLabels object

matchLabelKeys []string

mismatchLabelKeys []string

namespaceSelector object

matchExpressions []object

matchExpressions is a list of label selector requirements. The requirements are ANDed.

key string required

key is the label key that the selector applies to.

operator string required

operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.

values []string

matchLabels object

namespaces []string

topologyKey string required

drainTimeout string

DrainTimeout is how long a router-proxy replica keeps serving its in-flight requests after SIGTERM, so a rollout finishes the streams it holds instead of cutting them at the proxy's own default. It sets the proxy's --shutdown-timeout and the pod's terminationGracePeriodSeconds, which is this value plus a fixed 5s margin so the kubelet does not SIGKILL before the drain ends. A stream longer than this still ends at the drain deadline. The proxy's default of 30s applies when unset, and nothing is rendered on the pod template, so an operator upgrade does not roll existing proxies. Changing it changes the pod template, so it rolls the proxy pods, and a replica that drains stays Terminating for up to this long, delaying that roll. Must be between 1s and 1h.

image string

Image overrides the default router-proxy container image. Useful for air-gapped clusters that pin to an internal registry digest.

imagePullSecrets []object

ImagePullSecrets are passed through to the router-proxy pod spec.

name string

nodeSelector object

NodeSelector pins the router-proxy pod to nodes matching these labels. Use it when the proxy image is only present on specific nodes (for example a node-local registry) or to co-locate the proxy with its backends. Passthrough to the Pod spec.

podAnnotations object

PodAnnotations are merged into the router-proxy pod's metadata.annotations. The operator's own annotations (the config hash, and the prometheus.io scrape annotations when the operator emits them) win on collision. An annotation removed here is removed from the pods; annotations other tools add are kept.

podLabels object

PodLabels are merged into the router-proxy pod's metadata.labels, for example so label-based NetworkPolicies select the proxy. The operator's own labels (app, inference.llmkube.dev/model-router) win on collision, and the Deployment and Service selectors use only those, so changing PodLabels rolls the pods without touching a selector. A label removed here is removed from the pods; labels other tools add are kept.

quarantineDuration string

QuarantineDuration controls how long the proxy keeps a backend in the "skip" state after a 5xx or network error before allowing a half-open probe. A pod of an endpoint-resolution backend is quarantined after two such failures in a row, or at once when a stream goes silent past streamIdleTimeout. Default 15s when unset. Shorter windows make the proxy recover faster from transient blips; longer windows reduce probe load on genuinely-down upstreams. Tests can shrink this to sub-second to verify recovery without sleeping the full default.

replicas integer

Replicas is the desired number of router-proxy pods. Defaults to 1. The proxy is stateless for routing decisions; budget and SLO counters live in memory and reset on pod restart until the persistence feature lands. When any backend resolves to a ModelPool member, the operator pins this to 1 regardless of the value set here: ModelPool activation serializes swaps through a single in-process lock, so a second proxy replica would race it and thrash the shared GPU slot. Multi-replica pooled routers need cross-replica swap coordination (a lease), tracked as a follow-up.

format: int32

minimum: 1

maximum: 10

resources object

Resources sets the pod's compute resource requests and limits.

claims []object

Claims lists the names of resources, defined in spec.resourceClaims, that are used by this container. This field depends on the DynamicResourceAllocation feature gate. This field is immutable. It can only be set for containers.

name string required

Name must match the name of one entry in pod.spec.resourceClaims of the Pod where this field is used. It makes that resource available inside a container.

request string

Request is the name chosen for a request in the referenced claim. If empty, everything from the claim is made available, otherwise only the result of this request.

limits object

Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

requests object

Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

responseHeaderTimeout string

ResponseHeaderTimeout caps how long the proxy waits for the upstream to begin sending response headers. For non-streaming chat completions this is effectively a max-generation-time cap, because their headers arrive when generation completes. It does not bound a streaming response once its headers arrive; StreamIdleTimeout guards those. Default 120s when unset. Per-rule and per-backend timeouts (see RouterRule.Timeout and RouterBackend.Timeout) tighten this on a per-request basis but cannot extend it beyond this cap.

revisionHistoryLimit integer

RevisionHistoryLimit caps how many old ReplicaSets the proxy Deployment keeps for rollback. Unset keeps 3 old ReplicaSets; 0 keeps none. Useful to bound ReplicaSet buildup, since the proxy re-rolls on every config change.

format: int32

minimum: 0

runtimeClassName string

RuntimeClassName selects a Kubernetes RuntimeClass for the router-proxy pod, for example a sandboxed runtime such as gVisor or Kata. Maps directly to PodSpec.RuntimeClassName. A RuntimeClass with a scheduling section has its node selector and tolerations merged into the pod by the RuntimeClass admission controller, so no separate fields are needed. Unset leaves the pod spec unchanged.

serviceName string

ServiceName names the data-plane Service in front of the router-proxy pods, so clients can use a short in-cluster URL such as http://llmkube.<namespace>.svc.cluster.local/v1 (with endpoint.port 80). Unset keeps the generated name, <router>-router-proxy. Only that Service takes this name: the Deployment, ConfigMap, admin Service (<router>-router-proxy-admin), pod labels, and selectors keep their generated names, so changing it does not roll the proxy pods. status.endpoint follows the Service. A Service with this name that this ModelRouter does not control is never adopted or overwritten: the router reports RouterProxyNameTaken and keeps its previous Service. When the name changes (including back to unset), the previous Service this ModelRouter controls is deleted only after the new one exists. Clients of the old name must move, and a NodePort the cluster allocated or a LoadBalancer address is reassigned (a pinned endpoint.nodePort is carried over). Must be a DNS-1035 label and must not equal the admin Service's name. Ignored unless dataPlane is Proxy.

pattern: ^[a-z]([-a-z0-9]*[a-z0-9])?$

maxLength: 63

streamIdleTimeout string

StreamIdleTimeout caps how long a response body may go without a single byte from the upstream once its headers have arrived. A healthy stream emits bytes continuously; the longest gap is the prefill before the first content token, which can take minutes for a very long prompt on slow hardware. When the window passes with no bytes, the proxy aborts the request and quarantines that backend endpoint, so a wedged replica stops receiving traffic. Default 300s when unset; "0s" disables the guard.

tolerations []object

Tolerations are applied to the router-proxy pod, so it can be pinned next to tainted GPU nodes together with nodeSelector.

effect string

Effect indicates the taint effect to match. Empty means match all taint effects. When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute.

key string

Key is the taint key that the toleration applies to. Empty means match all taint keys. If the key is empty, operator must be Exists; this combination means to match all values and all keys.

operator string

Operator represents a key's relationship to the value. Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal. Exists is equivalent to wildcard for value, so that a pod can tolerate all taints of a particular category. Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators).

tolerationSeconds integer

TolerationSeconds represents the period of time the toleration (which must be of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately) by the system.

format: int64

value string

Value is the taint value the toleration matches to. If the operator is Exists, the value should be empty, otherwise just a regular string.

topologySpreadConstraints []object

TopologySpreadConstraints are passed through to the router-proxy pod, for example one replica per node with topologyKey kubernetes.io/hostname and the labels above. When replicas is more than 1 and a constraint uses whenUnsatisfiable: DoNotSchedule (or the affinity requires pod anti-affinity), the proxy rolls out with maxSurge 0 and maxUnavailable 1, because a surge pod could never be placed.

labelSelector object

LabelSelector is used to find matching pods. Pods that match this label selector are counted to determine the number of pods in their corresponding topology domain.

matchExpressions []object

matchExpressions is a list of label selector requirements. The requirements are ANDed.

key string required

key is the label key that the selector applies to.

operator string required

operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.

values []string

matchLabels object

matchLabelKeys []string

MatchLabelKeys is a set of pod label keys to select the pods over which spreading will be calculated. The keys are used to lookup values from the incoming pod labels, those key-value labels are ANDed with labelSelector to select the group of existing pods over which spreading will be calculated for the incoming pod. The same key is forbidden to exist in both MatchLabelKeys and LabelSelector. MatchLabelKeys cannot be set when LabelSelector isn't set. Keys that don't exist in the incoming pod labels will be ignored. A null or empty list means only match against labelSelector. This is a beta field and requires the MatchLabelKeysInPodTopologySpread feature gate to be enabled (enabled by default).

maxSkew integer required

MaxSkew describes the degree to which pods may be unevenly distributed. When `whenUnsatisfiable=DoNotSchedule`, it is the maximum permitted difference between the number of matching pods in the target topology and the global minimum. The global minimum is the minimum number of matching pods in an eligible domain or zero if the number of eligible domains is less than MinDomains. For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same labelSelector spread as 2/2/1: In this case, the global minimum is 1. | zone1 | zone2 | zone3 | | P P | P P | P | - if MaxSkew is 1, incoming pod can only be scheduled to zone3 to become 2/2/2; scheduling it onto zone1(zone2) would make the ActualSkew(3-1) on zone1(zone2) violate MaxSkew(1). - if MaxSkew is 2, incoming pod can be scheduled onto any zone. When `whenUnsatisfiable=ScheduleAnyway`, it is used to give higher precedence to topologies that satisfy it. It's a required field. Default value is 1 and 0 is not allowed.

format: int32

minDomains integer

MinDomains indicates a minimum number of eligible domains. When the number of eligible domains with matching topology keys is less than minDomains, Pod Topology Spread treats "global minimum" as 0, and then the calculation of Skew is performed. And when the number of eligible domains with matching topology keys equals or greater than minDomains, this value has no effect on scheduling. As a result, when the number of eligible domains is less than minDomains, scheduler won't schedule more than maxSkew Pods to those domains. If value is nil, the constraint behaves as if MinDomains is equal to 1. Valid values are integers greater than 0. When value is not nil, WhenUnsatisfiable must be DoNotSchedule. For example, in a 3-zone cluster, MaxSkew is set to 2, MinDomains is set to 5 and pods with the same labelSelector spread as 2/2/2: | zone1 | zone2 | zone3 | | P P | P P | P P | The number of domains is less than 5(MinDomains), so "global minimum" is treated as 0. In this situation, new pod with the same labelSelector cannot be scheduled, because computed skew will be 3(3 - 0) if new Pod is scheduled to any of the three zones, it will violate MaxSkew.

format: int32

nodeAffinityPolicy string

NodeAffinityPolicy indicates how we will treat Pod's nodeAffinity/nodeSelector when calculating pod topology spread skew. Options are: - Honor: only nodes matching nodeAffinity/nodeSelector are included in the calculations. - Ignore: nodeAffinity/nodeSelector are ignored. All nodes are included in the calculations. If this value is nil, the behavior is equivalent to the Honor policy.

nodeTaintsPolicy string

NodeTaintsPolicy indicates how we will treat node taints when calculating pod topology spread skew. Options are: - Honor: nodes without taints, along with tainted nodes for which the incoming pod has a toleration, are included. - Ignore: node taints are ignored. All nodes are included. If this value is nil, the behavior is equivalent to the Ignore policy.

topologyKey string required

TopologyKey is the key of node labels. Nodes that have a label with this key and identical values are considered to be in the same topology. We consider each <key, value> as a "bucket", and try to put balanced number of pods into each bucket. We define a domain as a particular instance of a topology. Also, we define an eligible domain as a domain whose nodes meet the requirements of nodeAffinityPolicy and nodeTaintsPolicy. e.g. If TopologyKey is "kubernetes.io/hostname", each Node is a domain of that topology. And, if TopologyKey is "topology.kubernetes.io/zone", each zone is a domain of that topology. It's a required field.

whenUnsatisfiable string required

WhenUnsatisfiable indicates how to deal with a pod if it doesn't satisfy the spread constraint. - DoNotSchedule (default) tells the scheduler not to schedule it. - ScheduleAnyway tells the scheduler to schedule the pod in any location, but giving higher precedence to topologies that would help reduce the skew. A constraint is considered "Unsatisfiable" for an incoming pod if and only if every possible node assignment for that pod would violate "MaxSkew" on some topology. For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same labelSelector spread as 3/1/1: | zone1 | zone2 | zone3 | | P P P | P | P | If WhenUnsatisfiable is set to DoNotSchedule, incoming pod can only be scheduled to zone2(zone3) to become 3/2/1(3/1/2) as ActualSkew(2-1) on zone2(zone3) satisfies MaxSkew(1). In other words, the cluster can still be imbalanced, but scheduler won't make it *more* imbalanced. It's a required field.

rules []object

Rules are evaluated in declaration order. The first matching rule wins. If no rule matches, the fallback is governed by DefaultRouteStrategy, ending at DefaultRoute. If nothing resolves, the request is rejected with HTTP 503.

failClosed boolean

FailClosed: when true, if no backend in Route.Backends is healthy or otherwise eligible, the router rejects the request with HTTP 503 rather than falling through to DefaultRoute or subsequent rules. This is the regulated-data gate: a fail-closed rule guarantees that matched requests are never served by any other route.

match object

Match groups all match conditions. All declared conditions must be true for the rule to fire (AND semantics). If Match is omitted the rule always matches (useful as a catch-all before DefaultRoute).

dataClassification []string

DataClassification matches if the inbound request carries any of these classifications. The classification is the value of the request header named by Policy.Classification.HeaderKey (x-llmkube-classification by default); the router does not classify the request body, so a request without the header matches no dataClassification rule. Common values: "public", "internal", "confidential", "pii", "phi".

headers object

Headers performs exact-match equality on inbound HTTP headers (case-insensitive header name comparison).

latencySLOMs integer

LatencySLOMs is a P95 first-token-latency target in milliseconds. When set, if the rolling P95 for the primary backend exceeds this value the rule promotes its declared fallback. Honored only by the "primary-fallback" strategy.

format: int32

minimum: 1

models []string

Models matches against the OpenAI-style "model" field in the request body. Glob patterns are supported (e.g. "qwen3-*").

requiredCapabilities []string

RequiredCapabilities filters backends. The rule only matches if at least one backend in Route.Backends advertises every listed capability, and on dataPlane: Proxy only those backends serve the rule's requests, under every strategy. A backend that lacks one is never tried, not even as a fallback.

taskComplexity string

TaskComplexity matches the inbound complexity hint (header x-llmkube-task-complexity).

enum: simple, moderate, complex

name string required

Name is used in audit logs and metrics labels.

pattern: ^[a-z0-9][a-z0-9-]{0,62}$

route object required

Route is the action taken when this rule matches.

backends []string required

Backends is an ordered list of RouterBackend.Name values. For the "primary-fallback" strategy, the first entry is the primary and subsequent entries are tried in order on failure. For "weighted", traffic is distributed across all entries by Backend.Weight. For "shadow", the first entry serves the response (falling back in order as primary-fallback does) and the entries after the one that served receive mirrored copies for evaluation only; it needs at least two.

minItems: 1

poolActivation string

PoolActivation decides what the proxy does when a backend in Backends is a ModelPool member that is not resident and the resident member is busy. "Wait" (default) holds the request open until the incumbent drains and the target loads, bounded by the pool's swapBudget. "IfIdle" only starts a swap when the incumbent is idle; while it is busy the pooled backend is skipped immediately and the request falls through to the next entry in Backends, so a rule such as [preferred-model, resident-model] serves on whichever is warm instead of queueing behind in-flight work. Non-pooled backends and same-model requests are unaffected.

enum: Wait, IfIdle

strategy string

Strategy selects how multiple backends are used. On dataPlane: Proxy, weighted picks a backend per request in proportion to its weight among the rule's backends that are healthy, not quarantined and advertise every required capability; a request that fails on it fails over to another backend of the rule picked the same way, never the same one twice. A weighted rule there cannot name two members of one ModelPool, since they share one GPU slot and would swap on every other request. shadow serves the client exactly as primary-fallback does and sends a copy of each request, marked x-llmkube-shadow: true, to the rule's other healthy backends after the one that served; their responses are discarded, their failures never quarantine them, and they are not charged to budgets. Copies are best effort: at most 4 run at once per backend and the rest are dropped and counted. An external backend that is a shadow target receives, and bills, every request. A shadow target cannot be a ModelPool member. dataPlane: Gateway rejects shadow. On dataPlane: AgentGateway only primary-fallback compiles, and a rule routes to its first healthy backend: when that backend's InferenceService has no ready replica, the next reconcile moves the rule to the next backend. Failover there follows replica readiness, not individual request errors.

enum: primary-fallback, weighted, shadow

timeout string

Timeout caps how long the proxy waits for the upstream to begin sending response headers on dispatches matched by this rule. When set it overrides RouterBackend.Timeout and the proxy default. Resolution order at dispatch time: rule.timeout || backend.timeout || proxy default. Useful for tightening regulated-data rules (sub-10s strict fail-fast) or extending long-reasoning rules (120s+).

status object

status defines the observed state of ModelRouter

activeRules integer

ActiveRules is the count of rules that successfully validated against current backend state.

format: int32

agentGateway object

AgentGateway reports the observed state of dataPlane: AgentGateway exposure: whether the InferencePool / HTTPRoute / model-extraction AgentgatewayPolicy reconciled against the referenced agentgateway Gateway, and the resolved endpoint. nil in Proxy and Gateway modes. Also surfaced via the AgentGatewayReady condition.

endpoint string

Endpoint is the agentgateway address clients send OpenAI requests to, resolved from the referenced Gateway. Empty until the pool is ready.

poolReady boolean

PoolReady indicates the generated InferencePool(s) reconciled against the referenced agentgateway Gateway.

backends []object

Backends reports the resolved address and current health of every declared backend.

address string

Address is the resolved upstream URL the router-proxy dispatches to. For local backends this is the InferenceService's cluster URL; for external backends it is the provider's base URL.

healthy boolean

Healthy reflects the most recent probe result.

lastProbeTime string

LastProbeTime is when the proxy last completed a health probe for this backend.

format: date-time

message string

Message provides extra context, especially when Healthy is false (e.g. "InferenceService not Ready", "Secret missing key ANTHROPIC_API_KEY").

name string required

Name matches RouterBackend.Name.

tier string

Tier mirrors RouterBackend.Tier for convenience.

budgetUtilization []object

BudgetUtilization summarises current budget consumption.

name string required

Name matches BudgetSpec.Name.

tokensUsed integer

TokensUsed is the rolling-window token count.

format: int64

usdUsed string

USDUsed is the rolling-window estimated cost in USD.

utilization string

Utilization is the fraction of the budget consumed, 0.0 to 1.0. When both MaxTokens and MaxUSD are set this is the maximum of the two utilizations.

conditions []object

conditions represent the current state of the ModelRouter resource. Standard condition types: - "Validated": the spec passed static validation - "BackendsReady": all referenced backends are reachable and healthy - "Available": the router-proxy is serving traffic - "Degraded": at least one backend is unhealthy but the router can still serve other routes - "GatewayReady": (dataPlane: Gateway) the gateway resources reconciled

lastTransitionTime string required

lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.

format: date-time

message string required

message is a human readable message indicating details about the transition. This may be an empty string.

maxLength: 32768

observedGeneration integer

observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance.

format: int64

minimum: 0

reason string required

reason contains a programmatic identifier indicating the reason for the condition's last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty.

pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$

minLength: 1

maxLength: 1024

status string required

status of the condition, one of True, False, Unknown.

enum: True, False, Unknown

type string required

type of condition in CamelCase or in foo.example.com/CamelCase.

pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$

maxLength: 316

endpoint string

Endpoint is the in-cluster URL clients should hit. Populated once the router-proxy Service is ready.

gateway object

Gateway reports the observed state of dataPlane: Gateway exposure: whether the AIGatewayRoute (and its backing Backend / AIServiceBackend / BackendTrafficPolicy) reconciled, and the resolved gateway endpoint. nil in Proxy mode. Also surfaced via the GatewayReady condition.

authEnabled boolean

AuthEnabled indicates a SecurityPolicy enforcing JWT authentication was compiled for this route (ModelRouter policy.auth.jwt). Set by the ModelRouter dataPlane: Gateway path; false when no auth is configured.

endpoint string

Endpoint is the gateway address clients send OpenAI requests to. Set by the ModelRouter dataPlane: Gateway path (resolved from the referenced Gateway); empty for the InferenceService path.

modelName string

ModelName is the resolved model-name match value clients send as the OpenAI "model" string to reach this InferenceService through the gateway. Set by the InferenceService path; empty for ModelRouter (which fronts many model names).

routeReady boolean

RouteReady indicates the AIGatewayRoute (and its backing Backend + AIServiceBackend) were reconciled successfully against the gateway.

lastUpdated string

LastUpdated is the timestamp of the last status reconciliation.

format: date-time

phase string

Phase is a coarse summary of the router's state. Possible values: Pending, Provisioning, Ready, Degraded, Failed.

enum: Pending, Provisioning, Ready, Degraded, Failed

No matches. Try .spec.agentGatewayRef for an exact path