Export view data to Amazon Kinesis

Stream every completed video view from FastPix into an Amazon Kinesis data stream in your own AWS account, then process the records with whatever pipeline you already run. Use streaming exports when you want view data in near real time rather than waiting for the daily CSV export, for example to feed a live alerting system, a warehouse, or a per-view workflow.

Before you begin

Make sure you have the following:

• A FastPix account on a paid plan. Streaming exports aren’t available on the free plan.
• An AWS account, and permission in it to create a Kinesis data stream, attach a resource policy to that stream, and (if you encrypt the stream with your own KMS key) edit that key’s policy.
• The FastPix role ARN shown on your streaming exports settings page. You name this role in your stream policy so FastPix can write to your stream.

How streaming exports work

FastPix writes each completed view into your stream with a PutRecord call. You own the stream and whatever reads it. FastPix only writes, using a permission you grant and can revoke at any time.

No credentials are exchanged. Only two values move between you and FastPix, and neither is a secret:

FastPix can’t read your stream, list your streams, delete anything, or change your retention. Writing to one stream is the only permission it holds.

Set up the data stream

Create an on-demand Kinesis data stream. On-demand capacity scales automatically, so you don’t have to size shards. Provisioned mode also works, but if you under-provision, FastPix is throttled and views arrive late.

Using the AWS console, go to Kinesis > Data streams > Create data stream, give the stream a name (for example video-views), select On-demand capacity mode, and create it.

Using the AWS CLI:

aws kinesis create-stream \
  --stream-name video-views \
  --stream-mode-details StreamMode=ON_DEMAND

Note the stream ARN. You paste it into FastPix later. In the console it’s on the stream’s Details tab with a copy icon; from the CLI it’s in the describe-stream-summary response.

arn:aws:kinesis:eu-west-1:123456789012:stream/video-views

Grant FastPix write access

Attach a resource policy to the stream that names the FastPix role from your export settings and allows it to write.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "FastPixStreamingExports",
    "Effect": "Allow",
    "Principal": {
      "AWS": "arn:aws:iam::<FASTPIX_ACCOUNT_ID>:role/fastpix-export-prod-eu"
    },
    "Action": ["kinesis:PutRecord", "kinesis:PutRecords"],
    "Resource": "arn:aws:kinesis:eu-west-1:123456789012:stream/video-views"
  }]
}

Using the console, go to Kinesis > Data streams > your stream > Data stream sharing > Create policy (or Edit if one exists), paste the JSON, and save.

Using the CLI, save the JSON as grant-fastpix-write.json, then apply and verify it:

aws kinesis put-resource-policy \
  --resource-arn arn:aws:kinesis:eu-west-1:123456789012:stream/video-views \
  --policy file://grant-fastpix-write.json
aws kinesis get-resource-policy \
  --resource-arn arn:aws:kinesis:eu-west-1:123456789012:stream/video-views

PutRecord sends one record and PutRecords sends a batch. Both are listed so batching can be enabled later without every customer editing their policy again. The policy allows writing to this one stream and nothing else.

NOTE: The role ARN belongs to FastPix’s AWS account, not yours. You aren’t creating it, you’re naming it in your policy. Copy it from your export settings page. A stream can hold only one sharing policy, so if yours already has one, add the statement above to it rather than replacing what’s there.

If your stream is encrypted with your own KMS key

Skip this section if you didn’t turn on server-side encryption, or if you use the AWS-managed key (aws/kinesis), whose policy you can’t edit.

With a customer-managed key, the stream policy alone isn’t enough. The key’s own policy must authorize the same FastPix role, or the write fails while your stream permission looks correct. Add this statement to the key policy:

{
  "Sid": "FastPixStreamingExports",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::<FASTPIX_ACCOUNT_ID>:role/fastpix-export-prod-eu"
  },
  "Action": "kms:GenerateDataKey",
  "Resource": "*"
}

Using the console, go to KMS > Customer managed keys > your key > Key policy > Edit, paste the statement into the Statement array, and save.

Using the CLI, fetch the current policy, add the statement to the Statement array, and put it back:

aws kms get-key-policy --key-id <KEY_ID> --policy-name default \
  --query Policy --output text > key-policy.json
# add the statement to the Statement array, then:
aws kms put-key-policy --key-id <KEY_ID> --policy-name default \
  --policy file://key-policy.json

Inside a key policy, "Resource": "*" means this key only. It doesn’t widen access beyond it.

IMPORTANT: If a write fails with a KMS error, the stream permission isn’t the problem, the key is. This is the most common way the setup goes wrong: the stream policy is correct, and the key policy is what blocks it. If you use the AWS-managed key and the connection test returns a KMS error, that key’s policy does not allow a cross-account write. Recreate the stream with a customer-managed key and add the statement above.

Set up a consumer

Set up whatever reads the stream before you enable the export, or the first records age out of your retention window before anything reads them. FastPix never touches this side. Common choices:

• Kinesis Data Firehose into S3 or Redshift, with no code, if you want the data in a warehouse.
• A Lambda trigger, for per-view processing.
• The Kinesis Client Library, for a long-running consumer that handles sharding and checkpointing.

Whatever you build, it needs to do two things: process only view records, and upsert on view_id. The example below is deliberately plain Python to show the shape rather than a framework.

import json, boto3
kinesis = boto3.client("kinesis", region_name="eu-west-1")
STREAM  = "arn:aws:kinesis:eu-west-1:123456789012:stream/video-views"
def handle(record):
    envelope = json.loads(record["Data"])
    # Not every record is a view. A connection_test arrives when someone
    # validates the export, and new data_type values can appear without a
    # version bump. Skip anything you don't recognize.
    if envelope.get("data_type") != "video_view":
        return
    view = envelope["data"]
    # Delivery is at-least-once. view_id is the stable identity of a view;
    # event_id is unique per delivery attempt, so it won't deduplicate a
    # retry. Upsert on view_id or your view counts drift upward quietly.
    db.upsert("video_views", key="view_id", row=view)

Most consumers use Firehose, a Lambda trigger, or the KCL rather than reading shards directly, because those handle sharding and checkpointing for you.

Finish setup in FastPix

Copy the stream ARN from the console Details tab, or from the CLI:

aws kinesis describe-stream-summary --stream-name video-views \
  --query StreamDescriptionSummary.StreamARN --output text

Paste it into Data Exports > Streaming exports in FastPix and save. There’s no separate region field, because the region is already in the ARN.

arn:aws:kinesis:eu-west-1:123456789012:stream/video-views

Saving stores the address and sends nothing yet. Select Run connection test to write one real record into your stream and start delivery. If it fails, you see the actual AWS error straight away.

On the Streaming exports tab, open the Service menu and choose Amazon Kinesis Data Streams.

FastPix then shows the role to grant, a ready-to-use resource policy, and the field where you paste your stream ARN.

NOTE: A workspace has one streaming destination at a time. To switch to a different stream or service, delete the current destination and set up the new one. The address can’t be edited in place.

What a view record looks like

Each record is a partition key (the view_id) and a JSON body. The body is a stable envelope wrapping a per-type data object, so a new event type can be added later without changing anything you already parse.

partition key:  6db4638e-2e97-43bd-a4b1-d88c00a17c6e     (the view_id)
data:
{
  "schema_version": "export_v1",
  "data_type":      "video_view",
  "event_id":       "evt_01K3T9WQ2M7YB4XC8HRN5ZQVD1",
  "workspace_id":   "9e655f67-20b5-4963-8bc7-1058a716faff",
  "view_id":        "6db4638e-2e97-43bd-a4b1-d88c00a17c6e",
  "emitted_at":     "2026-09-02T10:14:02.318Z",
  "data": {
    "video_title":   "test video",
    "view_start":    "2026-09-01T10:51:14.965Z",
    "view_end":      "2026-09-01T10:53:34.824Z",
    "watch_time_ms": 118568,
    "qoe_score":     0.9995538499937602,
    "city":          "Hyderabad",
    "os_name":       "MacOS",
    "cdn":           "cloudflare"
  }
}

This is byte-for-byte what a Google Cloud Pub/Sub customer receives, so your parser doesn’t change if you move between clouds. The full data object carries the same per-view fields as the daily CSV export. Timestamps are RFC 3339 in UTC, always. Viewer IP address and user agent aren’t exported.

What may change without a version bump

Handle the data correctly

Four rules cover the cases that trip up most integrations.

Filter on data_type. When you save your configuration, FastPix writes a connection_test record into your stream. It’s a real record your consumer receives. Process only records where data_type is video_view, and ignore any type you don’t recognize, because new ones can be added without a version bump.

Read identifiers from the body. A Kinesis record has no attribute mechanism, so every identifier lives in the JSON. view_id is both the partition key and a body field; event_id, workspace_id, and data_type are in the body only. Parse the record to route or deduplicate it.

Upsert on view_id, never insert. Delivery is at-least-once, so you’ll receive duplicates. If Kinesis redelivers a record your consumer didn’t checkpoint, the event_id repeats. If FastPix retries after a transient failure, the event_id is new but the view_id is the same. view_id is the stable identity of a view, so upsert on it. Inserting produces duplicate rows.

Don’t assume order. FastPix guarantees no ordering. Kinesis orders records within a shard, and the partition key is view_id, so records may appear to arrive in a meaningful order, but that isn’t a promise: it doesn’t survive resharding, doesn’t hold across shards, and can change without notice. Sort by view_end if you need chronological order.

NOTE: Watch retention and capacity. Kinesis keeps records for 24 hours by default. If your consumer falls behind your retention window, those records are gone, and draining that backlog is on your side. If you use provisioned mode and under-provision shards, FastPix is throttled and retries with backoff, so views arrive late rather than being lost. On-demand mode avoids this.

If your export was paused

FastPix pauses a destination when writing fails in a way retrying can’t fix, such as a revoked policy, a deleted stream, or a KMS key that no longer admits FastPix.

Views that arrive while the export is paused aren’t delivered, and aren’t sent later once it’s fixed. Delivery resumes from the moment it’s restored. FastPix records which views were missed, so support can tell you exactly what you didn’t receive and over what window. The daily CSV export still covers those days, so running it alongside streaming gives you a backfill for any gap.

If your plan changes

Moving to a free plan stops delivery, but your destination configuration is kept. Returning to a paid plan resumes delivery automatically, with nothing to set up again.

Troubleshooting

What’s next

• Export view data to Google Cloud Pub/Sub to stream the same records into Google Cloud instead.
• Export raw view data to pull daily CSV files instead of streaming.
• What Video Data do we capture for the full per-view field reference.