← Back
Home › Systems › Communication and media

transcoding-farm

video transcoding farm

YouTube, ffmpeg clusters

SystemTechniques · 5Primitives · 30 transcoding-farm DAG workflow executiondistributed · medium: Run a graph of dependent tasks by releasing each one when its inputs are done, longest chain first, and skip any task whose input hashes are unchanged.›Durable job queuedistributed · medium: Persisted tasks that workers lease for a visibility timeout, ack on success and retry with backoff; at-least-once execution that survives worker and queue crashes.›Chunked parallel encodingmedia · medium: Split video at keyframes, encode chunks in parallel, stitch with rate consistencyVideo transcoding laddermedia · medium: Turn one source into resolution x bitrate x codec renditions chosen from measured rate-quality points, cut on aligned keyframes so players can switch mid-stream.›Video rate controlmedia · medium: CBR/VBR/CRF rate control with VBV buffer model and lookaheadExponential backoff with jittersystems · low: Retry after delays that grow geometrically and are randomized, so failed clients spread out instead of hammering a recovering service in synchronized waves.›Leasesdistributed · low: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.›Idempotency keysfinance · low: Client-chosen unique key per logical operation, stored with a request fingerprint and the final response, so retries replay the first result instead of acting twice.›Media containersmedia · low: MP4/ISO BMFF, fragmented MP4/CMAF, Matroska and MPEG-TS mux/demuxScene cut detectionmedia · low: Detect shot changes from frame-difference statisticsPerceptual quality metrics (PSNR, S…media · low: Full-reference scores that say how bad a compressed picture looks: PSNR (pixel error), SSIM (structure), VMAF (learned fusion), and when each one lies.›Object storagecloud · low: S3-compatible blob storage at cents per GB-month with 11 nines durability; pick by egress price, request price and latency, not by storage price alone.›Hardware video accelerationmedia · low: GPU/ASIC encode and decode APIs (NVENC, VideoToolbox, VA-API) with zero-copy surfacesResumable uploadsnetworking · low: Open an upload session, send the body in offset-addressed chunks, and after any failure ask the server how much it committed and continue from there.›Bin packingoptimization · low: Pack items of given sizes into the fewest fixed-capacity bins; sort descending, then first-fit, gives at most 11/9 of optimal and runs in O(n log n).›Topological sortgraphs · low: Order the vertices of a DAG so every edge points forward; the core of build systems, schedulers and dependency resolution, with cycle detection for free.›Content-addressed storagestorage · low: Store each blob under the hash of its bytes: immutable, deduplicated by construction, self-verifying on every read, and composable into Merkle trees.›Dead-letter queuedistributed · low: A side queue for messages that keep failing: after N delivery attempts or a permanent error, park them with the reason so the main queue keeps moving.›Write-ahead logstorage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.›GOP structuremedia · low: How an encoder orders I, P and B frames into groups of pictures: the knob behind seekability, compression, latency and stream switching.›Image resampling filtersgraphics · low: Resize an image by convolving with a kernel (box, triangle, cubic, Lanczos) whose width scales with the shrink factor, done as two separable 1-D passes with precomputed weights.›+ 14 more primitives

Similar systems

Share the most building blocks with this one.

distributed-cron 18 %video-vod 18 %remote-build-execution 16 %digital-asset-management 16 %ci-server 16 %