Automatic broadcast recovery and stability improvements for Cloud Playout

Cloud Playout broadcasts now recover automatically after infrastructure restarts and rolling updates, alongside stability and billing-accuracy improvements.

What’s new:

• Automatic broadcast recovery: after a service restart or rolling update, an in-progress broadcast is detected and restarted automatically, so a channel heals on its own instead of staying down until manual intervention.

• Startup recovery reconciler: on startup, active broadcasts are rediscovered and auto-recovered, ended broadcasts are reconciled to idle, and already-running or future-scheduled broadcasts are left untouched.

• Recovery verification and health visibility: a recovered broadcast is checked for live output before it is reported healthy, and broadcast pipeline health and recovery status are available for monitoring.

Improvements:

• Safer rolling updates: a shutting-down instance releases only its own resources and drains in-flight work first, so it can’t overwrite the status of a broadcast that another instance just recovered.

Fixes:

• Accurate encoding-minute billing across restarts: encoding minutes are flushed on shutdown so usage is accounted for accurately across restarts. The flush is idempotent and never double-counts.

• Overlay crash guard: malformed, under-sized overlay frames are dropped before they can crash the pipeline, keeping channels running.

• Overlay and stream stability: overlay timestamps are re-based and realigned with the main video to fix overlay drift, plus hardening against stuck ingest streams.

• Failed broadcast starts surface an error: a broadcast that fails to start now reports an error to playout-manager instead of failing silently.