Media Infrastructure
Conduit
Internally, FMS
A media store built around a simple observation: most of what you keep, nobody is looking at. It tiers storage by what is actually being used.
- 3
- Storage tiers
- 29
- File types accepted
- 1,787
- Lines of JavaScript
- 10 GB
- Max upload
hot, warm, cold — by last access
10 image, 13 video, 6 subtitle
across 22 files
per file
What it is
Inside Conduit
Conduit is where our applications put media. Images, video, whatever a user uploads — it takes the file, works out what it actually is, files it somewhere it can be found again, and hands back a URL. It has no interface of its own; it is a service other things talk to. ruinsempire.com runs its video system on it, and univerge.life, still in build, uses it for avatars, banners, nameplates and profile effects.
The idea it was built around is that most stored media is not being looked at. A profile picture uploaded two years ago costs exactly as much to keep on fast storage as one uploaded this morning, and almost nobody ever asks for it again. So every file carries a storage tier — hot, warm or cold — alongside the time it was last read, indexed together, which makes “everything that has gone quiet” a single query. Keep the hot tier small and it stays both fast and cheap. The access tracking runs today, on every image and every video served. The job that acts on it and physically moves files between tiers is the part still to build.
Video turned out to be its own problem, and most of the recent work is there. The question worth answering is “can a browser play this as it stands?”, because the lazy answer — re-encode everything — is enormously wasteful. Conduit probes each file and checks container, video codec and audio codec against what browsers actually support. If all three pass, the original is served untouched, with range requests so seeking works. Only the failures get repackaged into HLS, and even then, anything already encoded as H.264 is copied into segments rather than re-encoded.
The single longest piece of work was a two-second bug. Subtitles rendered early, consistently, on everything transcoded. The cause is that ffmpeg’s transport-stream muxer does not start video at timestamp zero, but the subtitle tracks it extracts do — two clocks, quietly disagreeing. Conduit now probes the first segment it produces, reads where the video really starts, and writes a timestamp map into each subtitle file so the two agree. A migration went back and repaired everything transcoded before the fix existed.
Built with
- Node.js
- Express 4
- MongoDB + Mongoose
- Multer
- ffmpeg / ffprobe
- HLS (VOD)
- WebVTT
Capabilities
What it actually does.
Hot, warm and cold
Every file carries a tier and a last-read timestamp, indexed together so the set that has gone quiet is one query. The point is to keep the hot tier small, because that is the storage you pay for twice — once in money and once in speed. Honest status: the tracking is live on every file served, the mover that acts on it is not built yet.
Storage that does not rot
Files are renamed to a UUID and filed under year, month, and two directory levels derived from that UUID’s hash, so no single directory ever accumulates millions of entries. The path is reconstructable from the filename alone; the recursive fallback exists only as a safety net.
Decide before working
Container, video codec and audio codec are each checked against what browsers genuinely support. Anything already playable is served as-is and costs no CPU at all. Anything already H.264 is stream-copied into segments rather than re-encoded, so a four-gigabyte source repackages in minutes instead of hours.
The two-second subtitle bug
ffmpeg’s transport-stream muxer does not start video at timestamp zero, but the WebVTT cues it extracts do — so every subtitle landed one to two seconds early. Conduit probes the first segment it produced for its real start time and injects a timestamp map into each subtitle file. A failed probe degrades to zero, which is a harmless no-op.
A migration that repaired the past
Finding the bug was only half of it. A backfill walks every video transcoded before the fix, re-probes it and patches the subtitles in place, skipping any already corrected. Idempotent, so it can be run again without thinking about it.
Multiple audio tracks, and why that forces a repackage
Conduit repackages some files a browser could already play perfectly well — because browsers cannot reliably switch audio tracks inside a plain MP4. A dual-language file therefore becomes HLS with one rendition per track, exactly one marked default, preferring whichever the source nominated.
A queue that survives being killed
One transcode at a time, deliberately, because ffmpeg will otherwise take the whole machine. Jobs are tracked by database id rather than in memory, so on restart it re-enqueues anything left in pending or processing — that is, anything killed mid-encode.
Subtitles added after the fact
Upload an .srt to a video transcoded months ago and Conduit converts it, applies the same timestamp alignment, and surgically rewrites the existing playlist to reference it. The player cannot tell it from a track that was embedded all along, and nothing is re-encoded.
Related