ag-ui and litellm, explained for network engineers

i set out to invent my own ai protocol for a homelab project. then i asked the one question that saved me months.

so i’ve been brewing a little ai stack in the homelab — a gpu box (skynet), a web frontend (parthenon), and a desktop app (athena). somewhere in there i decided the frontend and backend needed a protocol to talk to each other, and i got excited about designing one from scratch. i’ve never rolled my own protocol before. sounded like a fun weekend.

then i asked the question every engineer should ask before building infrastructure: “has someone already standardized this?”

they had. twice over. here’s what i learned, explained the way i actually understood it — through networking analogies — because i’m a network engineer, not an ai researcher.

first: a protocol is not a program ✦

this trips people up, so let’s nail it down. you have run a web server. you have never “installed http.” you installed nginx, and you use a browser, and the two agree on the http standard — a document describing how they talk. http is the grammar; nginx and the browser are programs that speak it.

keep that split in your head. everything below is either a standard (a grammar) or an implementation (a program that speaks it). the whole point of a standard is that anybody’s program can talk to anybody else’s.

ag-ui = http for ai chat apps

the first thing i was about to reinvent already has a name: ag-ui, the agent-user interaction protocol.

it’s the standard language an app and an ai backend use over a live connection: streaming text as the model types, “the ai wants to run a tool,” “please approve this before i do it,” “here’s a file.” exactly the messages a chat/agent ui needs.

my homemade version already existed in my app as a pile of custom websocket messages i’d invented. ag-ui is just the published, standard version of the same thing — so any compliant app can talk to any compliant backend, the way any browser can load any website.

the part that sold me: it has a built-in, standardized flow for tool approval. my agent can edit files and run commands, and i gate those behind an approve/reject prompt. ag-ui models this natively — the agent pauses, the app shows an approval card, you say yes or no (and can even edit the arguments first), and the agent resumes. i was about to hand-build all of that. it’s in the spec, with working examples.

and critically: you’re not forced to use their library. there are official sdks (typescript, python) and community ones (rust, go, java, c++), but you could implement the events yourself — same as writing an http server from scratch. the sdk is a convenience, not a cage.

litellm = haproxy, but for models ⚗️

this one’s even more familiar if you’ve run a load balancer.

you know haproxy: one front door, many backends behind it, health checks, routing rules. litellm is that, for ai models. you point everything at one address; it forwards each request to whichever model you configured — something local on your own gpu, or claude, or openai — and papers over their differences so everything only has to speak one dialect (openai’s). on top of that it adds api keys, rate limits, budgets, and usage metering per user.

that’s it. it’s a reverse proxy for llms. they literally call it an “ai gateway.”

two things it does that matter a lot:

• load-balancing across multiple copies of a model — two gpus, and requests spread across both automatically.
• fallback chains — “if my local model is down, use claude instead.” you configure it once. when the gpu box is offline, chat keeps working.

mcp = usb for tools

quick one, because i’d already adopted it: mcp (model context protocol) is the standard way an ai plugs into tools — run a command, read a file, hit an api. any tool that speaks mcp works with any ai, the way any usb device works on any computer. my tool gateway is basically the hub everything else plugs into.

how it all fits together

here’s the mental model that finally clicked. four layers, three standard plugs between them:

app  (web frontend / desktop app)
  │   ag-ui            ← "http for ai chat": chat, tools, approvals
  ▼
agent orchestrator  (the brain: runs the agent loop, tools, image jobs)
  │   openai dialect
  ▼
litellm  (model router: one door, load-balance, fallback, keys)   ← lives on the always-on box
  │   openai dialect
  ▼
model runners  (local gpu models · claude · openai)

the app talks to the brain in ag-ui. the brain fetches model answers through litellm, and reaches tools through mcp. the models themselves — local or cloud — sit at the bottom.

a subtlety i had backwards at first: i kept calling the middle two layers “the backend,” as if they were one thing. they’re not. the orchestrator is the agent brain; litellm is just the switchboard in front of the raw models. litellm fronts the models, not the whole app — because it only understands “give me a completion,” not “run this agent with these tools.”

the gotcha i almost shipped

i wanted auto-failover: gpu box goes down → automatically use claude. litellm does exactly that.

but if litellm runs on the gpu box, and the gpu box dies, litellm dies with it — and now there’s nothing left to do the failing over. the router has to outlive the thing it’s failing away from.

one more caveat: failing over from a free local model to a paid cloud model has a cost cliff. litellm’s per-key budgets let you cap that, so a flaky gpu can’t quietly run up a bill.

“but shouldn’t i build my own?”

this was my real hesitation — not “does it work,” but “am i giving something up by not building it?” here’s the logic that settled it, and it’s the kind of risk table worth stealing:

look at that last row:

“the worst case of adopting equals the best case of building my own — minus the design work.” ⚗️

except the hard parts, like that tool-approval flow, are already solved and battle-tested. because these projects are mit-licensed, i can keep a copy forever. there’s no outcome where adopting leaves me worse off than starting from scratch.

the one real risk isn’t “the repo vanishes” — it’s “the industry picks a different standard in a few years.” and you neutralize that with one design habit: wrap the protocol behind a single adapter layer. your app’s core logic talks to your own internal interface; a thin shim translates to the protocol on the wire. swap protocols later? you rewrite one shim, not the whole app. same reason you put gear behind a consistent management interface instead of scattering vendor cli commands through all your automation.

licenses, in sixty seconds

if you’re going to depend on this stuff, check the licenses first. i did:

• ag-ui, litellm, mcp, ollama, the tool gateway — all mit. do anything.
• comfyui (my image backend) — gpl-3.0. the scary one, until you understand it.

gpl only bites when you (a) ship the software to someone and (b) weld it into your own code. run it as a separate service you talk to over the network and you’re fine — exactly like a closed-source app querying a gpl mysql database over a socket, which the whole industry does without becoming gpl. so the rule is simply: never bundle comfyui’s source into my app; always run it as its own local service. gpl stays quarantined to comfyui itself.

the takeaway

i didn’t get to invent a protocol. i’m a little sad about that — it’s filed on the someday-list, right next to approximately forty other cursed ideas. but here’s the reframe that made me happy again:

adopting the boring, standardized plumbing is what frees you to build the interesting parts. i’m not re-milling the screws, i’m skipping straight to building the cabinet. the chat wire-format is solved; my time goes to the stuff that’s actually mine — the multi-gpu scheduler, the desktop app that’s also a compute node, the image pipeline.

ask “does this exist yet?” first. sometimes the answer costs you a fun weekend project. more often it saves you six months of maintaining something the rest of the world already agreed on. 🩷

the three standards: ag-ui · litellm · mcp. all mit-licensed.