Export view data to Google Cloud Pub/Sub

Stream every completed video view from FastPix into a Google Cloud Pub/Sub topic in your own project, then read the records into BigQuery, a push endpoint, or a client library. Use streaming exports when you want view data in near real time rather than waiting for the daily CSV export, for example to feed a live alerting system, a warehouse, or a per-view workflow.

Before you begin

Make sure you have the following:

• A FastPix account on a paid plan. Streaming exports aren’t available on the free plan.
• A Google Cloud project with billing enabled. Pub/Sub doesn’t work without it.
• Permission to create topics and edit IAM in that project (roles/pubsub.admin or owner).
• The FastPix service account email shown on your streaming exports settings page. It looks like fastpix-export-prod-eu@fastpix-prod.iam.gserviceaccount.com. This email isn’t a secret. Knowing it grants no access; permission exists only because you grant it below.

How streaming exports work

FastPix publishes each completed view into your topic. You own the topic and the subscription. FastPix only publishes, using a permission you grant and can revoke at any time.

No credentials are exchanged. Only two values move between you and FastPix, and neither is a secret:

FastPix never receives your password, key, or any credential, and it can’t read your subscription. It can only write to your topic.

Set up the topic

Create a Pub/Sub topic. Using the Google Cloud console, go to Pub/Sub > Topics > Create topic, give the topic an ID (for example fastpix-views), and leave Add a default subscription ticked, which creates the subscription for you.

Using the gcloud CLI:

gcloud pubsub topics create fastpix-views --project=YOUR-PROJECT

Topic naming rules: 3 to 255 characters, starts with a letter, may contain a-z A-Z 0-9 - _ . ~ + %, no /, and can’t start with goog.

IMPORTANT: Pick the name carefully. Once the export is live, renaming or deleting the topic stops delivery and requires FastPix to update your configuration by hand.

Grant FastPix publish access

Grant the FastPix service account the Pub/Sub Publisher role on the topic, not on the whole project.

Using the console, go to Pub/Sub > Topics > your topic > Permissions > Add principal, paste the FastPix email, choose the role Pub/Sub Publisher, and save.

Using the CLI:

gcloud pubsub topics add-iam-policy-binding fastpix-views \
  --project=YOUR-PROJECT \
  --member="serviceAccount:fastpix-export-prod-eu@fastpix-prod.iam.gserviceaccount.com" \
  --role="roles/pubsub.publisher"

Confirm the binding:

gcloud pubsub topics get-iam-policy fastpix-views --project=YOUR-PROJECT

You should see:

bindings:
- members:
  - serviceAccount:fastpix-export-prod-eu@fastpix-prod.iam.gserviceaccount.com
  role: roles/pubsub.publisher

Grant roles/pubsub.publisher and nothing broader. FastPix doesn’t need, and shouldn’t be given, any other permission.

NOTE: If the console rejects the email with a message that it must belong to an active Google account, the principal picker can’t resolve a service account from a different organization. That’s a limitation of the picker, not the permission. Use the gcloud command above instead, which makes the identical binding.

If your organization enables Domain Restricted Sharing, you may see One or more users named in the policy do not belong to a permitted customer. Your org policy administrator must allow FastPix’s organization before the export can be set up. Ask them about constraints/iam.allowedPolicyMemberDomains.

Create the subscription

Skip this if you left Add a default subscription ticked when you created the topic.

Using the console, go to Pub/Sub > Subscriptions > Create subscription, set a subscription ID (for example fastpix-views-sub), select the topic fastpix-views, and choose Pull delivery.

Using the CLI:

gcloud pubsub subscriptions create fastpix-views-sub \
  --topic=fastpix-views --project=YOUR-PROJECT

WARNING: Create the subscription before you tell FastPix to enable the export. A Pub/Sub topic doesn’t store anything. On publish it copies the message into every subscription that exists at that moment, then forgets it. With no subscription, the message is discarded immediately with no error: FastPix logs “published”, you receive nothing, and the message can’t be recovered.

Finish setup in FastPix

Copy the full topic path from the console Topic name column, or from the CLI:

gcloud pubsub topics describe fastpix-views --project=YOUR-PROJECT \
  --format="value(name)"

Either way you end up with:

projects/YOUR-PROJECT/topics/fastpix-views

Paste it into Data Exports > Streaming exports in FastPix and save. Saving stores the address and sends nothing yet. Select Run connection test to publish one real message to your topic and start delivery.

On the Streaming exports tab, open the Service menu and choose Google Cloud Pub/Sub.

FastPix then shows the service account to grant, a ready-to-use gcloud command, and the field where you paste your topic path.

Until the connection test passes, the destination is saved but sends nothing, and there is no on/off switch to flip.

After the connection test passes, the destination shows a Delivering status with the validated topic and schema version.

NOTE: A workspace has one streaming destination at a time. To switch to a different topic or service, delete the current destination and set up the new one. The address can’t be edited in place.

Read your data

Set up a consumer before you enable the export, so the first messages have somewhere to go. There are three common approaches, easiest first.

Stream into BigQuery with no code

Best if you want the data in your warehouse. Create a BigQuery subscription on the topic:

gcloud pubsub subscriptions create fastpix-views-bq \
  --topic=fastpix-views \
  --project=YOUR-PROJECT \
  --bigquery-table=YOUR-PROJECT:analytics.fastpix_views \
  --write-metadata

--write-metadata adds subscription_name, message_id, publish_time, and an attributes column alongside data. The five message attributes arrive together in attributes as JSON, so the identifiers are reachable without parsing the body:

SELECT
  JSON_VALUE(attributes, '$.view_id')   AS view_id,
  JSON_VALUE(attributes, '$.event_id')  AS event_id,
  JSON_VALUE(data, '$.data.watch_time_ms') AS watch_time_ms
FROM `YOUR-PROJECT.analytics.fastpix_views`;

Deduplicate on view_id and filter out connection_test messages. There’s an equivalent for Cloud Storage using --cloud-storage-bucket.

Push to your endpoint

Pub/Sub POSTs each message to an HTTPS endpoint you own, with no client library and no long-running process:

gcloud pubsub subscriptions create fastpix-views-push \
  --topic=fastpix-views \
  --project=YOUR-PROJECT \
  --push-endpoint=https://your-app.example.com/fastpix-views

Return 2xx to acknowledge a message. Any other status causes a retry.

Pull with a client library

Full control over how each message is handled.

from google.cloud import pubsub_v1
import json
subscriber = pubsub_v1.SubscriberClient()
path = "projects/YOUR-PROJECT/subscriptions/fastpix-views-sub"
def callback(message):
    view_id = message.attributes["view_id"]
    envelope = json.loads(message.data)
    upsert_view(view_id, envelope["data"])   # upsert, never insert
    message.ack()
subscriber.subscribe(path, callback=callback).result()

const {PubSub} = require('@google-cloud/pubsub');
new PubSub().subscription('fastpix-views-sub').on('message', msg => {
  const viewId = msg.attributes.view_id;
  const envelope = JSON.parse(msg.data.toString());
  upsertView(viewId, envelope.data);        // upsert, never insert
  msg.ack();
});

Subscriber subscriber = Subscriber.newBuilder(
    ProjectSubscriptionName.of(projectId, "fastpix-views-sub"),
    (message, consumer) -> {
        String viewId = message.getAttributesMap().get("view_id");
        JsonNode envelope = MAPPER.readTree(message.getData().toStringUtf8());
        upsertView(viewId, envelope.get("data"));   // upsert, never insert
        consumer.ack();
    }).build();
subscriber.startAsync().awaitRunning();

To inspect messages by hand, pull from the subscription in the console (Pub/Sub > Subscriptions > your subscription > Messages > Pull, with acknowledgement left unchecked so messages aren’t deleted as you read them), or from the CLI:

gcloud pubsub subscriptions pull fastpix-views-sub --project=YOUR-PROJECT --limit=5

What a message looks like

Each message carries attributes you can read without parsing the body, plus a JSON body.

The attributes are useful for filtering and deduplication:

The body is UTF-8 JSON, a stable envelope wrapping a per-type data object. In the REST API and gcloud --format=json the body appears base64-encoded, which is encoding, not encryption, and the console decodes it for you.

{
  "schema_version": "export_v1",
  "data_type": "video_view",
  "event_id": "evt_01M0SMDPZGAQEECSJCH895YMHD",
  "workspace_id": "9a20af9c-3556-4e26-a6ec-d269056d971a",
  "view_id": "91cf737b-b684-44d2-8bfc-69f2058098ac",
  "emitted_at": "2026-08-24T10:18:05.412Z",
  "data": {
    "video_title": "Dental Billing 101: Mastering Efficiency and Profitability",
    "view_start": "2026-08-24T10:17:12.793Z",
    "view_end": "2026-08-24T10:17:40.136Z",
    "watch_time_ms": 27342,
    "qoe_score": 0.921535226295831,
    "country_code": "IN",
    "city": "Hyderabad",
    "os_name": "MacOS",
    "browser_name": "Chrome",
    "cdn": "cloudflare"
  }
}

The example above is trimmed for readability. The full data object carries 117 fields, matching the columns of the daily CSV export. Fields with no value for a view are omitted, not sent as null. Two identifiers you might expect inside data, workspace_id and view_id, are one level up on the envelope instead.

This is byte-for-byte what an Amazon Kinesis customer receives, so your parser doesn’t change if you move between clouds. Timestamps are RFC 3339 in UTC with three decimal places. Viewer IP address and user agent aren’t exported.

Field groups

What may change without a version bump

New optional fields may be added inside data, and new values may appear in existing fields, so tolerate both and ignore fields you don’t recognize. A new data_type may appear, so ignore ones you don’t know. Renames, removals, and type changes never happen silently: they require export_v2 and a migration window where both versions are published. The envelope (schema_version, data_type, event_id, workspace_id, view_id, emitted_at) and the RFC 3339 UTC timestamp format don’t change.

Handle the data correctly

Four rules cover the cases that trip up most integrations.

Create the subscription before the export is enabled. A topic with no subscription discards messages silently, as noted above. This is the mistake people actually make.

Filter on data_type. When you set up or re-test the export, FastPix publishes one connection_test message to prove the grant works before accepting your configuration. It arrives in the same envelope as a view, with a different type. Because data_type is a Pub/Sub attribute as well as a body field, you can drop it without deserializing: check attributes.data_type == "video_view". A consumer that reads data.view_id unconditionally fails on this message, and it arrives before any real data.

Upsert on view_id, never insert. Delivery is at-least-once, so you’ll receive duplicates. If Pub/Sub redelivers an unacknowledged message, the event_id repeats. If FastPix re-sends a view, the event_id is new but the view_id is the same. Upsert on view_id, which is the stable identity of a view. Inserting produces duplicate rows.

Don’t assume order. FastPix guarantees no ordering, on every destination, so a pipeline written against that keeps working if you move. FastPix doesn’t use Pub/Sub ordering keys, because they serialize publishing and slow delivery for everyone. Completed views are independent and carry their own timestamps, so sort by view_end if you need chronological order.

NOTE: Pub/Sub keeps unacknowledged messages for 7 days by default. If your consumer is down longer than that, the backlog is dropped. Raise it with --message-retention-duration if you need more.

If your export was paused

FastPix stops publishing to a topic when it gets a permanent error, such as PERMISSION_DENIED, NOT_FOUND, UNAUTHENTICATED, or INVALID_ARGUMENT. This protects other customers from a broken destination. The most common causes are the IAM binding from the grant step being removed, and the topic being renamed or deleted.

Views published while the export is paused are lost, not queued, and not replayed once it resumes.

To restart it, fix the cause (usually restore roles/pubsub.publisher on the topic), then open Data Exports > Streaming exports in your FastPix dashboard and select Re-check and resume. FastPix re-checks the grant before resuming, and delivery starts from the next view to complete. The daily CSV export still covers those days, so running it alongside streaming gives you a backfill for any gap.

If your plan changes

Moving to a free plan stops delivery, but your destination configuration is kept. Returning to a paid plan resumes delivery automatically, with nothing to set up again.

Troubleshooting

What’s next

• Export view data to Amazon Kinesis to stream the same records into AWS instead.
• Export raw view data to pull daily CSV files instead of streaming.
• What Video Data do we capture for the full per-view field reference.