Category definition

Vertical
Inference

System intelligence beyond the model.

Vertical Inference is the architecture in which the application, the model runtime and live inference state become one programmable system.

Ordinary code governs how intelligence retrieves, reasons, acts and continues while it is underway. A token API makes inference callable; Vertical Inference makes intelligence application-programmable.

Lloyal is the platform for building and deploying this architecture — from local applications to frontier compute.

The inversion

Endpoint architecture

Application logic surrounds detached model requests. Working state is serialised across a provider boundary. The endpoint returns outputs; the application reconstructs intelligence around them.

Vertical Inference

Model execution becomes resident application state. Ordinary code programs topology, evidence, tools, policy, authority, completion and continuity while intelligence is alive.

A serving stack schedules requests.
Vertical Inference schedules lineages.

Not local inference alone. Not a request graph moved in-process. A general model becomes a governed domain capability through application code that programs its live execution.

Named July 2026 · first published 10 August

Three properties.

We call the architecture vertical inference: model execution becomes resident application state, and ordinary code programs topology, evidence, tools, policy, authority, completion and continuity while the intelligence is still alive. How Lloyal turns the model’s working memory into agents on the GPU

Resident means the model’s state lives in the same process as the code that governs it, not in a datacentre that holds it on terms you do not set. Addressable means a lineage of that state is an object with a position, a parent and a lifetime, which application code forks, steers, admits evidence into, and ends. Governed while alive means the decision happens before the token becomes state, not after the answer arrives.

Each is load-bearing. Resident without addressable is a hosted model. Addressable without governed is a tracing library. Governed after the fact is the state of the art.

The closest familiar shape is a game engine. Game code does not call a physics service and wait for a result — it runs inside the frame, reading and writing simulation state while the loop advances. Vertical Inference puts application code in that position with respect to inference.

Where the code runs

The application programs inference while it is underway.

An agent system today holds two memories. The transcript belongs to the application. The KV cache belongs to a remote GPU, and a modern serving stack may well keep it after the response completes — but it keeps it as a cache of its own, evicted on its own policy and never addressable from your code.

Almost everything frustrating about agent systems follows from that one split, and reuse underneath does not close it. Continuity is a transcript replayed into a prompt, whether or not the prefix is recomputed. Concurrency is N conversations that cannot see one another, however well the router places them. And the bill multiplies per agent, because each agent is its own request.

The bill is the smallest of it. A trajectory in progress is unreachable from outside it. A client inspects a finished output and accepts or rejects it. It cannot fork that trajectory, route evidence into it, steer what it may conclude, or rule that a particular continuation should not become the state the next step inherits. By the time application code runs, the reasoning is over.

Two panels. Left, without Lloyal: application policy prepares a request and evaluates a response around a model that fans a shared prefix into four independent trajectories; modern serving can batch requests and reuse matching prefixes, but those trajectories remain server-managed. Right, with Lloyal: application policy governs reasoning while it is underway, forking, adding evidence, steering, concluding and pruning across four named trajectories over one shared live state, with no post-hoc validation loop because policy has already acted. — The same model, two degrees of control — policy acting *around* the call, and policy acting *through* it. What earns a place in a context already spoken for is decided per candidate. Prefix caching moves the left panel’s cost; it does not move the boundary.

Architecture

Five layers.

Shaping inference to the workload happens at five depths, each with its own unit of work. A harness — the application you write — sits above all five and is yours; these are the layers it runs on. The layers are not a product line — they are one line drawn through a stack that is usually cut horizontally, which is the whole reason the category needed a name.

Layer	Availability	Unit of work
Runtime	Available now	The lineage and the lease — a tree of owned lifetimes governing a tree of live inference state, scheduled by an admission controller that admits, refuses or reclaims a lease mid-run. Structured concurrency over Git-like branching of attention state, addressed from ordinary TypeScript.hdk hdk/agents
Bindings	Available now	The native backend — kernel primitives surfaced to the host language, with prebuilt binaries for thirteen platform and GPU combinations and the accelerator chosen at runtime rather than install time.lloyal.node
Kernel	Available now	KV cells in VRAM — KV tenancy, atomic inference state branching, Continuous Tree Batching, 3-way semantic merging. A header-only C++20 kernel, compiled into the host binary.liblloyal
Device	In development	The equipment — native loops on manufactured hardware. Launch partners, 2026–27 production cycles.
Silicon	Research	Hardware that assumes the tree. A fork already costs no cells and a group of branches already settles in one dispatch — the economics hold today, on general accelerators. Silicon is where the shape stops being expressed and starts being assumed.

What runs today

The lineage

An agent is a branch with intent. The branch is the ground truth — the cells, the position, the sampler. The agent is the interpretation layer that gives that state meaning and decides what to do next. A harness is a tree of owned lifetimes governing a tree of live inference state, and the contract between them is narrow: state inherits, judgment does not. A fork inherits the KV prefix and the spine header, never the policy, guards or grants.

The lease

An agent is a tenancy lease on live model state, and the application owns the lease. Our own kernel header states the problem: slots are abundant, leases are scarce. The counter that makes concurrency safe is the one that makes it programmable — headroom, a soft floor where spawning stops, a hard floor where work stops before the next decode. Not telemetry. An admission controller. Adaptive compute follows from the same counter: whole lines of work are pruned on judgement rather than windows compacted on arithmetic.

Cells in GPU memory

Kernel in the operating-system sense. llama.cpp is a library we link, not a server we call — and liblloyal is a header-only C++20 overlay on it, an INTERFACE target with no binary of its own, compiled into the same process. Neither of them is the runtime. The runtime is the layer above them, and it is what a harness is written against: structured concurrency over Git-like branching of attention state. A single physical cell in VRAM can carry many owners at once, which is why a fork costs a bit rather than a buffer — branch, fork, and a merge that is semantic rather than textual.

What a kernel provides

Capability	What it adds to llama.cpp
KV tenancy	Leases over `seq_id`s, with the invariant that a vacant sequence is always clean. llama.cpp executes a sequence; it has no notion of who holds one.
Atomic inference state branching	KV sequence, sampler chain, grammar parse state, metrics, logits and bias cloned as one unit — generation-counted, so slot reuse cannot alias.
Continuous Tree Batching	N branches bin-packed into a single tick, O(1) in branch count. Not requests from unrelated callers — branches of one tree that share their history physically.
3-way semantic merging	Two branches’ live distributions combined into a third — `merge_logits(dst, experts, alpha)`, additive in log space, with `alpha` weighting how far the experts pull. The experts are different KV states of the same model, so their own histories shape the next token. Contrastive decoding applied to branches rather than to separate models.

Agents are rows

The last two rows are where the category stops being an argument about software architecture. To the hardware there is no dimension for sequences at all — there are token rows, each tagged with an owner, and one forward pass covers them. And 3-way semantic merging is only available to something holding both histories at once.

A merge from inside the process

The second of those is unavailable from outside it. Two branches investigate the same question under different framings and reach different conclusions. The usual move is to take both finished answers and ask a third model to reconcile them — a summarisation problem, with everything that invites. Here the reconciliation happens a level down: merge_logits(dst, experts, alpha) combines the branches’ live distributions additively in log space, where alpha sets how hard the experts pull. Nothing is summarised, because nothing has been emitted. What is merged is what the model is about to say, rather than what it has said.

The name

Why ‘Vertical’ Inference?

Lloyal named the axis in July 2026, and the name is the argument. Horizontal inference widens one layer of the stack across many independent sequences; Vertical Inference drives a single line of work down through every layer of it — the application’s own code, the runtime that schedules it, the kernel that owns its cells, and the hardware beneath them. Both are scaling strategies. They scale different things.

Every production inference system optimises the horizontal axis, and it is a good axis. Continuous batching re-decides batch membership at every decode step, so a finished sequence frees its slot immediately and the accelerator never waits on the slowest request. Orca named it iteration-level scheduling; vLLM and SGLang call it continuous batching, TensorRT-LLM calls it in-flight batching, LMDeploy calls it persistent batching. It is universal, and the state of the art reaches much further than a shared prefix: NVIDIA Dynamo tracks KV blocks across an entire GPU fleet in a radix tree, scores every arriving request for cache overlap, routes it to the worker already holding the most of it, and will disaggregate prefill from decode and move the KV between their VRAMs to get it there. The axis is still width. What is scheduled is still a request.

System	What it batches	Unit of work
vLLM · TGI · TensorRT-LLM	Independent sequences, re-selected every decode step. Paged KV removes the fragmentation that capped batch width.	The request
SGLang	The same, plus prefix reuse across requests. RadixAttention keeps finished KV in a radix tree, matches new prompts against it automatically, and evicts least-recently-used leaves under pressure.	The request
llama.cpp server	Slots, each owning a sequence, decoded together in one unified batch. A slot re-uses its own cached prefix when the next request overlaps it.	The request
NVIDIA Dynamo	Requests placed across a whole fleet by KV overlap, with prefill and decode disaggregated and cache moved between workers’ VRAM to serve them.	The request
Vertical Inference	Branches of one tree, bin-packed into a single tick, O(1) in branch count. They share history physically because one forked from another deliberately.	The branch

Whose tree it is

None of this changes the axis, and the reason is ownership rather than cleverness. However much a serving stack knows about your KV — and Dynamo knows where every block in the fleet lives — that knowledge is infrastructure: built for you, invisible to you, evicted or relocated as its own capacity dictates. You cannot name a node, hold one, fork from one on purpose, or decide when it dies. In Vertical Inference the tree is the interface. A branch is an object your code holds, under a lease it was granted, for a lifetime it decides.

Where they separate

The two axes come apart most clearly at the merge. SGLang can fork a prompt into parallel branches and rejoin them as text. Vertical Inference rejoins them before there is any text, in the distributions themselves. Horizontal systems make many conversations cheap. Vertical Inference makes one conversation programmable.

The platform

The Lloyal platform in five stacked bands. Application surfaces: CLI, desktop, mobile and web, joined by one application contract. The harness in ordinary TypeScript, covering procedure, orchestration, retrieval, tools, policy, authority, trace and completion. Signed agent apps distributed as portable capabilities. Live attention: one shared live model state decoded once and inherited everywhere, arranged as fan-out, chain or DAG, with context admission and policy. Deployment: on-device mobile, local laptop, on-premises appliance, cloud and frontier compute. — Surfaces to deployment — the runtime, bindings and kernel layers, seen from the application side. Placement is a deployment decision; the category continues below this diagram’s last band.

The extension model

The third band is where capability arrives. An Ability installs into a harness and runs inside the model’s execution rather than behind a network boundary, so it can see which agent called it, what that caller already knows, and what else is running — and can fork that state into new agents.

On the same terms

Abilities are signed and distributed through a verified channel, first- and third-party alike. A capability that can fork live inference state is only safe to install if its provenance is checkable, which is why the channel is part of the architecture rather than a storefront bolted to it.

Where it goes next

Device — the equipment

Below the kernel the unit stops being a cell and becomes a machine somebody manufactures. We published this direction in the HDK months ago: a native C++ runtime for equipment makers, running multi-agent observe–orient–decide–act loops on board, taking sensor fusion in and dispatching actuation as tool calls. No AI box, no networking overhead. In development with launch partners, co-engineering toward 2026–2027 production cycles.

Silicon — research

The line does not stop at the kernel. On general accelerators this architecture already runs at the cost it claims: a fork shares its parent’s cells rather than copying them, so branching is free, and a whole group of branches settles in a single dispatch whatever its width. That is where N agents for the cost of one comes from, and it is true today. Silicon is not where those economics get rescued — it is where the shape of the workload stops being something the layers above express and becomes something the hardware assumes. Lloyal Labs joined the NVIDIA Inception program in August 2026, and the kernel one layer above it is already written.

In practice

The architecture is in production.

Ten agents shared one 239 GB model on two RunPod B200s for 82 minutes, and the session cost $16 in total — not per agent. That is the easiest property to measure and the least interesting. The larger claim is that ordinary application code governs live inference state while it runs, and the code for that is public.

“Cost scales with the size of the cache, not the number of agents.”Zhen Lu, CEO of RunPod — details

Residency	The model runs in the same process as the code that governs it — offline as a single on-device binary, as a private service on an appliance inside your network, or on frontier compute, all three today. Placement is a deployment decision, not a port. None of this is expressible against a hosted endpoint, because an endpoint’s unit of sale is the request.
The source	The work sits in public repositories under the Functional Source License. `liblloyal` is the kernel. `agent-pool.ts` is the scheduler. The architecture is documented at docs.lloyal.ai, and the shortest way in is `npx lloyal-ai new`.