Detect skips and rewatches in your video data - and find your most replayed moments

Viewers rarely watch a video start to finish. They skip ahead, jump back, and rewatch the parts that matter to them. Every one of those moves ends up in your view data, and they answer two useful questions: what are viewers looking for, and what do they come back to?

This article turns that record into two maps - where viewers skip to, and what they rewatch - and then into clips of your most replayed moments.

Here's the catalog-level view you'll build. Skips land across the whole timeline; rewatches cluster in a few sections. Those clusters are your most replayed moments.

How the data flows

fastpix-video-data-sample-export.csv: https://drive.google.com/file/d/1iMRkZEryzGc0_7GdAc_eTzd4Xluz-ZCu/view?usp=sharing

80 real (anonymized) views from an online course catalog, exported from the FastPix Video Data API. 136 columns per view, including the full playback event stream. Every snippet below runs against it. No FastPix account yet? The sample works without one - or start free with $25 credit (no credit card) and run this on your own views instead.

Part 1: Get view data flowing from your player

Playback happens in your player. FastPix Video Data captures every interaction from inside it: it dedupes the beacons, stitches events into sessions, adds geo and device data, and computes quality scores, across millions of concurrent views when you need it. What you export is one clean row per view.

Already collecting views with FastPix? Skip to Part 2. If you're new here, the whole setup takes about ten minutes.

Step 1: Add the player (or add monitoring to the one you have)

The FastPix Player has the Video Data beacon built in. Add the embed and views start flowing to your workspace:

<script src="https://unpkg.com/@fastpix/[email protected]/dist/player.js"></script>

<fastpix-player playback-id="YOUR_PLAYBACK_ID" stream-type="on-demand"></fastpix-player>

Keeping your existing player? Add the monitoring SDK for your stack instead: hls.js, video.js, or dash.js on web, plus native mobile SDKs. Same data either way. The sample dataset in this article was collected from an hls.js player.

import fastpixMetrix from "@fastpix/data-core";

fastpixMetrix.tracker(videoElement, {
  hlsjs: hls,
  Hls: Hls,          // both are required
  data: { workspace_id: "YOUR_WORKSPACE_KEY" },   // the only mandatory field
});

Step 2: Pass who's watching through custom dimensions

Worth adding before you move on: identity. With it, every view carries your user ID, and the skip and rewatch patterns below become per-learner and per-course.

fastpixMetrix.tracker(videoElement, {
  hlsjs: hls,
  Hls: Hls,
  data: {
    workspace_id: "YOUR_WORKSPACE_KEY",
    custom_2: `${user.id}::${user.email}`,  // who's watching
    custom_5: lesson.slug,                  // which lesson
    custom_9: course.title,                 // which course
  },
});

If custom_2 comes back null in your export, the usual cause is setting metadata after the tracker initialized. Pass it inside the tracker() call, not later.

Step 3: Play something, then export

Play a few videos. Open the Video Data dashboard and each playback session shows up as a row within seconds. Export from Views → Export CSV, or pull the same rows from the export API.

One row = one view, 136 columns. This article uses video_title, video_duration, and above all events: a JSON array of everything that happened during playback, each event carrying a playhead position (pt, ms into the video) and a wall-clock timestamp (vt):

[
  {"pt": 0,     "e": "viewBegin", "vt": 1776318843241},
  {"pt": 1,     "e": "playing",   "vt": 1776318844833},
  {"pt": 48483, "e": "pause",     "vt": 1776318893415}
]

Part 2: Build the skip and rewatch maps

The stack is chdb (embedded ClickHouse, pip install chdb) plus Plotly. The same SQL runs unchanged on clickhouse-local, self-hosted ClickHouse, or ClickHouse Cloud when you outgrow CSVs.

Step 4: Load the export

import chdb

views = chdb.query("""
    SELECT view_id,
           video_title,
           CAST(video_duration AS Int64) AS duration_ms,
           events
    FROM file('fastpix-video-data-sample-export.csv', CSVWithNames)
    WHERE video_duration > 0
""", "DataFrame")

Step 5: Detect skips and rewatches

A jump is playhead movement that didn't happen at playback speed. Between two consecutive events, if the playhead moved forward much faster than the wall clock, the viewer skipped ahead. If it moved backward, they went back to rewatch. No seek-event names needed, which matters because some players collapse or rename their seeking/seeked pairs.

import json

def jumps(events_json):
    """[(kind, from_ms, to_ms), ...] - 'skip' forward, 'rewatch' backward."""
    evs = [e for e in json.loads(events_json)
           if e.get("pt") is not None and e.get("vt")]
    out = []
    for a, b in zip(evs, evs[1:]):
        d_pt = b["pt"] - a["pt"]          # playhead movement (ms)
        d_vt = b["vt"] - a["vt"]          # wall-clock elapsed (ms)
        if b["vt"] - evs[0]["vt"] < 2000:
            continue                       # ignore resume-restores at view start
        if d_pt - d_vt > 3000:
            out.append(("skip", a["pt"], b["pt"]))
        elif d_pt < -1000:
            out.append(("rewatch", a["pt"], b["pt"]))
    return out

views["jumps"] = views["events"].map(jumps)

Two thresholds worth knowing about. The 2-second guard at the start exists because players with a resume feature restore the last position as one big forward jump; without the guard, every returning viewer looks like they skipped. And 3 seconds is the forward cutoff separating a real skip from tiny playhead corrections after buffering.

On the sample export this finds 288 skips and 53 rewatches - viewers skip ahead 5.4x more often than they go back.

Start using FastPix for free - $25 credit, no credit card required. Run this on your own videos in about ten minutes. Create your workspace →

Step 6: Map where the jumps land

Bucket every landing position into 5% segments of its video:

BUCKETS = 20
def seg(pos_ms, dur): return min(int(pos_ms / dur * BUCKETS), BUCKETS - 1)

skip_land, rewatch_land = [0]*BUCKETS, [0]*BUCKETS
for _, r in views.iterrows():
    for kind, f, t in r.jumps:
        (skip_land if kind == "skip" else rewatch_land)[seg(t, r.duration_ms)] += 1

Plot both as grouped bars (full Plotly code in the notebook) and you get the chart at the top of this page.

Step 7: Find each video's most replayed moments

Same idea, per video, rewatches only:

replayed = {}
for _, r in views.iterrows():
    for kind, f, t in r.jumps:
        if kind != "rewatch": continue
        b = seg(t, r.duration_ms)
        key = (r.video_title, r.duration_ms, b)
        replayed[key] = replayed.get(key, 0) + 1

for (title, dur, b), count in sorted(replayed.items(), key=lambda kv: -kv[1])[:5]:
    print(f"{title}: {b*5}-{b*5+5}% ({count} rewatches)")

If a video's biggest rewatch cluster is right at the start, treat it separately: those are mostly restarts and second looks at the opening, not a moment viewers studied. The interesting clusters are mid-video.

Part 3: What the sample data says

Viewers skip ahead 5.4x more often than they rewatch. 288 skips against 53 rewatches, and the skips land everywhere: 37% of them past the halfway mark, with a median jump of 10 seconds. Viewers are looking for specific moments without any navigation to help them. Chapters give them somewhere to click, and in-video search lets them type instead of drag the progress bar.

Rewatches cluster in a few sections. The mid-video clusters sit at 40 to 60% of the timeline: the orientation video has one at 40-45% (7 rewatches) and another at 55-60% (5), and the workshop recording has a pair around 40-50%. Whatever is at those timestamps is the content people came back for.

Two neighboring sections are usually one moment. The workshop's two clusters sit side by side, which usually means one explanation spanning the boundary. Merge them before clipping rather than cutting the moment in half.

Part 4: Turn the most replayed moments into clips

Your viewers already voted with the progress bar. This loop finds the sections they rewatch most and emits clip requests with timestamps, padded so the clip opens slightly before the moment:

def fmt(ms): return f"{ms//60000}:{(ms//1000)%60:02d}"

PAD = 15_000  # open the clip a little before the moment
for (title, dur, b), count in sorted(replayed.items(), key=lambda kv: -kv[1]):
    if count < 3 or b == 0: continue     # ignore restarts and weak signals
    start, end = int(b/BUCKETS*dur) - PAD, int((b+1)/BUCKETS*dur) + PAD
    # swap print() for the FastPix clip API call
    print(f"CLIP REQUEST: \"{title}\" {fmt(max(start,0))}-{fmt(min(end,dur))} "
          f"({count} rewatches here)")

Run against the sample export:

CLIP REQUEST: "Orientation: Welcome & Program Overview" 21:45-25:00 (7 rewatches here)
CLIP REQUEST: "Orientation: Welcome & Program Overview" 30:00-33:15 (5 rewatches here)
CLIP REQUEST: "Workshop: Applied Case Studies" 23:05-26:30 (3 rewatches here)
CLIP REQUEST: "Workshop: Applied Case Studies" 26:00-29:25 (3 rewatches here)

Wire the print into the instant clipping API and the staged clips land in your review queue. Other loops the same data supports:

The workflow cookbook covering these end to end is coming as a follow-up.

Add this to your weekly report

Most teams don't run this once; they watch it move. Add two lines to the report your content team already reads: the skip-to-rewatch ratio for videos published in the last 30 days, and the new most replayed sections with their clip requests. Run the notebook on the same Monday cron as your retention report. A full article on building that weekly video engagement report is coming.

How different teams use this

Running this on your own data

Point Step 4 at your own export. Two slices worth trying first: WHERE device_type = 'Mobile' to see whether touch controls change the skip pattern, and grouping by custom_9 (course) to find which courses make viewers search the most. If your skip counts come out near zero on real data, check that your beacon events carry pt values; a player that only reports wall-clock timestamps can't support jump detection.