Cloudflare Stream is a powerful broadcasting platform that, for many of our customers, just works. But what if you wanted to render dynamic annotations on a livestream or create an alternate version of a hosted video with burned-in subtitles? You would need to run a custom video pipeline.
Today, we’re releasing a new developer playground, Streamline, that demonstrates how you can build a system to deliver these bespoke video experiences on Cloudflare’s Developer Platform. We’ll walk you through how Streamline leverages Workers, Containers, and several media protocols to modify video — and immediately publish that output as livestream or new hosted video.
You’ll also have the opportunity to try it for your projects. A processing pipeline needs a durable, long-running environment that can run specialized, compiled code with predictable memory and CPU capacity. Video streams can run for minutes or hours, so the media process needs a lifecycle independent of the request that started it.
An application should be able to start a pipeline, send its input, inspect it, and stop it without needing to keep a single request open for the entire duration. Cloudflare provides the primitives we need. Containers are long-lived runtimes suitable for media processing.
Durable Objects help with orchestration. Finally, Workers are perfect for control signaling and monitoring. For Streamline, we built a media engine running in a Container to handle media processing in real-time.
The Container is controlled by a Worker exposing control, preview, and testing to an agent or user. Processing will continue even if the Worker disconnects. We've architected Streamline with modular components so that the media engine could be replaced with dedicated encoding products in the future.
Architecture A Streamline deployment consists of two components: the Media Engine , which handles media input/output and processing, and a controlling Application , which creates, configures, observes, and stops media sessions. Media Engine The Media Engine has two components: Controller. This is a control harness written in Go that implements an HTTP server, receives incoming requests, and translates them into operations that can be executed by the media engine.
Processor that performs the actual media processing. The current implementation uses FFmpeg, but that is an internal implementation detail rather than part of the user-facing API. The Media Engine is hosted in a Container, and handles all media input/output as well as processing.
It can pull RTMPS playback over the network from one Stream Live input and publish RTMPS output to another Stream Live input. It can pull a Cloudflare Stream HLS manifest and its segments to use hosted videos as input. It can accept video input from a source supplied by the controlling application, for example a webcam.
It can publish preview video over an outbound WebSocket to a Durable Object relay. An application that needs preview can connect to that relay through its own WebSocket. Application The application is built using Workers, and can be a full-stack browser application, an agent, or an embedded system.
It consists of: User interface (UI) including client logic, identity and access policy. This post uses a browser application as its concrete example, so it also includes a browser interface. Orchestrator coordinates the session, the Container lifecycle, and preview relay.
The orchestrator is implemented by a Durable Object. It is possible to run the system locally during development, in which case the container is just a local Docker instance and the Durable Object is not used: there is a single user, the controlling application does not require authorization for local access, and the video preview can connect directly to a WebSocket on localhost.
When these components are deployed to Cloudflare, an authorized user or agent can visit the Worker to start a new session. This spins up a new Streamline container if needed, manages its lifecycle automatically, exposes an API to perform a number of video manipulation operations, and routes inputs from and outputs back to Cloudflare Stream. Time for a technical deep dive on how the system works.
Container lifecycle and session management The controlling Worker application initiates a long-running media processing session. After starting the session, the application can disconnect and reconnect safely, while the Container continues processing until the controlling application stops it. We also include a maximum duration to ensure a session is always eventually closed down and can’t run indefinitely, even without external control.
While a media processing session is running, the container instance is unavailable for other applications to use. A Cloudflare Container will automatically sleep if it has not received any incoming requests since a defined interval. However, in our case, once the pipeline is running, it must continue even if the controlling application disconnects and it receives no requests.
We can implement this behavior by overriding the onActivityExpired() callback on the container. If the expiry time has not been reached, then we renew the activity, otherwise we destroy the container. API The HTTP server implemented by the Go harness and the Durable Object associated with the Container together define the low-level interface to the system.
However, we wanted to provide an abstraction over this, so the system is as agnostic as possible to who or what is controlling the session and any unnecessary details of the backend implementation. We implement this by exporting two packages from Streamline: @cloudflare/streamline/client Defines a high-level, session-based API. @cloudflare/streamline/ Exposes the Durable Object base class associated with the container.
This routes the API requests, implements the preview relay server described below, and provides hooks for security and access policy.
Originally published at blog.cloudflare.com
