AI Infrastructure
AI Relic
A self-hosted AI stack — an 80-billion-parameter coding model on our own GPU, and the coding environment built around it. Nothing leaves the machine.
- 7
- Services
- 256K
- Context window
- 80B
- Parameters
- ~27k
- First-party code
inference, memory, coding environment, images, arbitration, broker, presence
tokens, per session
mixture of experts, 3B active
lines across Python, Node and C++; over half is the coding environment
What it is
Inside AI Relic
AI Relic is what happens when you decide an AI assistant should not require anyone else’s datacentre. It runs on a single workstation — an NVIDIA RTX PRO 6000 Blackwell alongside an RTX 3080 Ti — and serves an 80B mixture-of-experts coding model with a 256,000-token context window over an OpenAI-compatible API. No request crosses the network unless we send it there.
It is not one program but seven cooperating services. An inference server. A persistent memory layer backed by Postgres and pgvector, with a sleep cycle that consolidates what it learned. An offline image generator. A GPU arbitration daemon. A broker that lets a small planning model and the large coding model collaborate on one shared document. A Discord Rich Presence publisher that needs no Discord client running. And the thing most of the code actually is: a coding environment.
That environment started as an agentic terminal — read files, write patches, run commands, step through a task — and grew an IDE inside itself. Press /code and the viewport becomes three regions: a file explorer, an editable file, and a pane where the model explains the line your cursor is sitting on. They are one pane rather than three because they share state no boundary should cross: the explorer’s selection opens the editor’s file, the editor’s cursor row is the explain pane’s entire input, and moving that cursor is what aborts the explanation already in flight.
The hardest problem here was not the model — it was tenancy. The coder’s weights occupy roughly 77 GiB and it claims most of what remains. The image generator wants the same card. The Arbitor settles it with leases, and when it refuses a request it names the exact command that would resolve the conflict. Its authority is deliberately asymmetric: it can move the image daemon, but it may never stop the coder, so nothing it decides on its own can cost you a six-minute reload.
Built with
- Python 3.12
- TensorRT-LLM
- CUDA 13.2
- Node.js 22 (ESM)
- C++20
- PostgreSQL + pgvector
- OpenMPI
- nginx + WireGuard
- systemd
Capabilities
What it actually does.
80B model, 256K context, entirely local
A mixture-of-experts coding model served in FP8 with CUDA graphs, chunked prefill and a paged cache sized for roughly three concurrent full-context sessions. The API is OpenAI-compatible, so existing tooling just points at it. Nothing leaves the machine.
An IDE that lives in the terminal
The /code view fills the viewport with an explorer, an editable file and an explanation pane. The explorer is deliberately flat rather than indented — two columns per level meant a file four directories deep had its name pushed off the right edge of a 34-column pane, which is the one thing a file list must never do.
The model explains the line you are on
The best-behaved feature in the project, because of what it refuses to do. There is one inference server and the human is usually waiting on it, so an explanation nobody asked for out loud never competes: while anything else is generating, it waits. Only the latest request is kept, so a long turn spent moving around resolves to exactly one call at the end, for the line the cursor actually stopped on — and the moment it moves again, the stream in flight is aborted rather than allowed to land under the wrong line number.
An editor written from scratch, for good reason
Node’s own line editor could not be used: it owns the cursor and writes to stdout whenever it likes, which in a full-screen layout means drawing straight through the transcript. So raw mode is taken for the whole session and one module owns every keystroke. Undo is stored as inverse operations rather than snapshots — a snapshot stack on a twenty-thousand-line file costs megabytes per keystroke, while the inverse of an insert is just the range to delete. And newlines are Ctrl-J, not Shift-Enter, because terminals transmit Shift-Enter identically to Enter and the binding would silently do nothing.
A text buffer you can test without a terminal
The buffer behind the editor touches no filesystem, no terminal, no screen and no timers. That purity is not tidiness — it is the only way an off-by-one in cursor arithmetic is ever actually caught, because the alternative is asserting against a pseudo-terminal.
Diffs rendered the way git renders them
Unified only, never side by side: at forty-five columns a split view gives you two useless twenty-character gutters. Two line-number columns, old and new, because one cannot answer “where was this before”. Syntax highlighting is switched off inside a diff on purpose — the add/remove colour carries the meaning and has to win.
An agent loop that could move to a browser unchanged
Eight tools — read, write, edit, bash, glob, grep, test, agents — behind a loop that yields a permission request and receives the decision back through the same channel, rather than taking an injected callback. The consequence is that the core serialises to a websocket frame without modification, and a consumer that cannot reply is failed closed rather than auto-approved. Sub-agents, a debate mode, and context compaction sit on top.
The Arbitor — a GPU lease broker
Two tenants that cannot coexist on one card. A tool declares its job and receives a lease, or a refusal that names the command which would fix it. It can relocate the image daemon but is structurally forbidden from touching the coder, so nothing it decides on its own can cost a six-minute reload.
Image generation that never touches the network
Model code, weights, tokenizers and the Python environment all live inside the daemon. It returns filesystem paths rather than base64, because the caller is on the same box.
Memory with a sleep cycle
Conversations and embeddings in Postgres with pgvector, streamed over a socket. Closing that socket aborts generation, so the GPU stops working on an answer with nowhere to go.
Planner and coder, brokered
The two models collaborate on one plan document through a delimiter protocol the phone already parses, so answers apply with no new client code. Delimiters appearing inside user text are defused, so writing about the feature cannot terminate a block early.
Related
Where it sits.
The launcher is a client of it — image generation, chat, and the GPU tenancy strip.
Ruins Launcher
Shipping over the air · build 25
An Android home screen that grew into a phone OS layer — and updates itself over the air from our own server.
Read more →