Turbo Pipeline Architecture

Help users make architecture decisions for Turbo pipelines — source types, data flow patterns, resource sizing, sink strategies, streaming vs job mode, dynamic table design, and multi-chain deployment.

Agent Instructions

Step 1: Understand the Requirements

Ask the user about:

1. What data? — Which chain(s), which events/datasets, historical or real-time only?
2. Where does it go? — Database, webhook, streaming topic, multiple destinations?
3. How much volume? — Single contract or all chain activity? How many events/sec?
4. Latency needs? — Real-time dashboards (sub-second) or analytics (minutes OK)?

Step 2: Recommend an Architecture

Use the patterns and decision guides below to recommend a pipeline architecture. Reference the templates in templates/ as starting points:

• templates/linear-pipeline.yaml — Simple decode → filter → sink
• templates/fan-out-pipeline.yaml — One source → multiple sinks
• templates/fan-in-pipeline.yaml — Multiple events → UNION ALL → sink
• templates/multi-chain-templated.yaml — Per-chain pipeline pattern

Step 3: Hand Off to Implementation Skills

After the architecture is decided, direct the user to:

• /turbo-pipelines — To build and deploy the YAML
• /turbo-transforms — To write the SQL transforms
• /secrets — To set up sink credentials

Reminder: When presenting complete pipeline YAML as part of architecture recommendations, validate it first with goldsky turbo validate. Templates in templates/ are structural patterns — any customized version must be validated before presenting to the user.

---

Source Types

Dataset Sources — Historical + Real-Time

Use type: dataset when you need to process historical blockchain data and/or continue streaming new data.

sources:
  my_source:
    type: dataset
    dataset_name: base.erc20_transfers
    version: 1.2.0
    start_at: earliest  # or: latest

Best for: Raw logs, transactions, transfers, blocks — anything where you need historical backfill or decoded event processing.

Available EVM datasets: raw_logs, blocks, raw_transactions, raw_traces, erc20_transfers, erc721_transfers, erc1155_transfers, decoded_logs

Non-EVM chains also supported: Solana (solana.*), Bitcoin (bitcoin.raw.*), Stellar (stellar_mainnet.*)

Kafka Sources — Real-Time Streaming

Note: Kafka sources are used in production pipelines but are not documented in the official Goldsky docs. Use with caution and contact Goldsky support for topic names.

Use type: kafka when consuming from a Goldsky-managed Kafka topic, typically for continuously-updated state like balances.

sources:
  my_source:
    type: kafka
    topic: base.raw.latest_balances_v2

Best for: Balance snapshots, latest state data, high-volume continuous streams.

Key differences from dataset sources:

• No start_at or version fields
• Optional fields: filter, include_metadata, starting_offsets
• Delivers the latest state rather than historical event logs

When to Use Which

| Scenario | Source Type | Why | | --------------------------------- | ----------- | -------------------------------------------- | | Decode contract events from logs | dataset | Need raw_logs + _gs_log_decode() | | Track token transfers | dataset | erc20_transfers has structured data | | Historical backfill + live | dataset | start_at: earliest processes history | | Live token balances | kafka | latest_balances_v2 is a streaming topic | | Real-time state snapshots | kafka | Kafka delivers latest state continuously | | Only need new data going forward | Either | Dataset with start_at: latest or Kafka |

---

Data Flow Patterns

Linear Pipeline

The simplest pattern. One source → one or more chained transforms → one sink.

source → transform_a → transform_b → sink

Use when: You have a single data source, single destination, and straightforward processing (decode, filter, reshape).

Example: templates/linear-pipeline.yaml — raw logs → decode → extract trade events → postgres

Resource size: s or m

Fan-Out (One Source → Multiple Sinks)

One source feeds multiple transforms, each writing to a different sink.

              ┌→ transform_a → sink_1 (clickhouse)
source ──────┤
              └→ transform_b → sink_2 (webhook)

Use when: You need different views or subsets of the same data going to different destinations — e.g., balances to a warehouse AND token metadata to a webhook.

Example: templates/fan-out-pipeline.yaml — one Kafka source → fungible balances to ClickHouse + all tokens to a webhook

transforms:
  fungible_balances:
    type: sql
    primary_key: id
    sql: |
      SELECT ... FROM latest_balances balance
      WHERE balance.token_type = 'ERC_20' OR balance.token_type IS NULL

  all_tokens:
    type: sql
    primary_key: id
    sql: |
      SELECT ... FROM latest_balances balance
      WHERE balance.token_type IN ('ERC_20', 'ERC_721', 'ERC_1155')

sinks:
  warehouse:
    type: clickhouse
    from: fungible_balances
    # ...
  webhook:
    type: webhook
    from: all_tokens
    # ...

Resource size: m (multiple output paths)

Fan-In (Multiple Events → One Output)

Multiple event types decoded from the same source, normalized to a common schema, then combined with UNION ALL into a single sink.

              ┌→ event_type_a ──┐
source → decode ┤                 ├→ UNION ALL → sink
              └→ event_type_b ──┘

Use when: You want a unified activity feed, combining trades, deposits, withdrawals, transfers, etc. into one table.

Example: templates/fan-in-pipeline.yaml — one raw_logs source → decode → multiple event-type transforms → UNION ALL → ClickHouse

Resource size: l (complex processing with many transforms)

Multi-Chain Fan-In

Multiple sources from different chains combined into a single output.

source_chain_a ──┐
source_chain_b ──┼→ UNION ALL → sink
source_chain_c ──┘

Use when: You want cross-chain analytics or a unified view across chains.

sources:
  eth_transfers:
    type: dataset
    dataset_name: ethereum.erc20_transfers
    version: 1.0.0
    start_at: latest
  base_transfers:
    type: dataset
    dataset_name: base.erc20_transfers
    version: 1.2.0
    start_at: latest

transforms:
  combined:
    type: sql
    primary_key: id
    sql: |
      SELECT *, 'ethereum' AS chain FROM eth_transfers
      UNION ALL
      SELECT *, 'base' AS chain FROM base_transfers

Resource size: m or l depending on chain count

---

Templated Multi-Chain Deployment

When you need the same pipeline logic across multiple chains, create separate pipeline files per chain rather than one multi-source pipeline. This gives you:

• Independent lifecycle (deploy/delete per chain)
• Independent checkpointing (one chain failing doesn't block others)
• Clearer monitoring per chain

Pattern: Copy the pipeline YAML and swap the chain-specific values:

| Field | Chain A (base) | Chain B (arbitrum) | | ------------------ | --------------------------------- | ------------------------------------- | | name | base-balance-streaming | arbitrum-balance-streaming | | topic | base.raw.latest_balances_v2 | arbitrum.raw.latest_balances_v2 | | Source key | base_latest_balances_v2 | arbitrum_latest_balances_v2 | | Transform SQL | 'base' AS chain | 'arbitrum' AS chain | | Sink table | base_token_balances | arbitrum_token_balances |

Example: templates/multi-chain-templated.yaml — shows the base chain version; duplicate for each chain.

When to use templated vs multi-source:

| Approach | Pros | Cons | | ----------------------- | --------------------------------------------- | ----------------------------------- | | Templated (per-chain) | Independent lifecycle, clear monitoring | More files to manage | | Multi-source (one file) | Single deployment, cross-chain UNION possible | Coupled lifecycle, harder to debug |

---

Resource Sizing

Each size doubles the previous tier's CPU and memory:

| Size | Workers | CPU Request | Memory | When to Use | | ----- | ------- | ----------- | ------ | -------------------------------------------------------------- | | xs | — | 0.4 | 0.5 Gi | Small datasets, light testing | | s | 1 | 0.8 | 1.0 Gi | Testing, simple filters, single source/sink, low volume (default) | | m | 4 | 1.6 | 2.0 Gi | Multiple sinks, Kafka streaming, moderate transform complexity | | l | 10 | 3.2 | 4.0 Gi | Multi-event decoding with UNION ALL, high-volume historical backfill | | xl | 20 | 6.4 | 8.0 Gi | Large chain backfills, complex JOINs (e.g. Solana accounts+transactions) | | xxl | 40 | 12.8 | 16.0 Gi | Highest throughput needs; up to 6.3M rows/min |

Rules of thumb from production pipelines:

• Simple filter + single sink → s (default, try this first)
• Kafka source + multiple sinks OR multiple transforms → m
• Raw log decoding + 5+ event types + UNION ALL → l
• Historical backfill of high-volume data → l or xl (can downsize after catch-up)
• Start small and scale up — defensive sizing avoids wasted resources

---

Sink Selection

Quick Reference

| Destination | Sink Type | Best For | | -------------------- | -------------------- | --------------------------------------------- | | Application DB | postgres | Row-level lookups, joins, application serving | | Real-time aggregates | postgres_aggregate | Balances, counters, running totals via triggers| | Analytics queries | clickhouse | Large-scale aggregations, time-series data | | Event processing | kafka | Downstream consumers, event-driven systems | | Serverless streaming | s2_sink | S2.dev streams, alternative to Kafka | | Notifications | webhook | Lambda functions, API callbacks, alerts | | Data lake | s3_sink | Long-term archival, batch processing | | Testing | blackhole | Validate pipeline without writing data |

Decision Flowchart

What's your primary use case?
│
├─ Application serving (REST/GraphQL API)
│  └─ PostgreSQL ← row-level lookups, joins, strong consistency
│
├─ Analytics / dashboards
│  ├─ Time-series queries → ClickHouse ← columnar, fast aggregations
│  └─ Full-text search → Elasticsearch / OpenSearch
│
├─ Real-time aggregations (balances, counters)
│  └─ PostgreSQL Aggregate ← trigger-based running totals
│
├─ Event-driven downstream processing
│  ├─ Need Kafka ecosystem → Kafka
│  └─ Serverless / simpler → S2 (s2.dev)
│
├─ Notifications / webhooks
│  └─ Webhook ← HTTP POST per event
│
├─ Long-term archival
│  └─ S3 ← object storage, cheapest for bulk data
│
├─ Just testing
│  └─ Blackhole ← validates pipeline without writing
│
└─ Multiple of the above
   └─ Use multiple sinks in the same pipeline (fan-out pattern)

PostgreSQL Aggregate Sink

The postgres_aggregate sink is uniquely suited for real-time running aggregations (balances, counters, totals). It uses a two-table pattern: a landing table that receives raw events, and an aggregation table maintained by a database trigger.

sinks:
  token_balances:
    type: postgres_aggregate
    from: transfers
    schema: public
    landing_table: transfer_events
    agg_table: account_balances
    primary_key: id
    secret_name: MY_POSTGRES
    group_by:
      account:
        type: text
      token_address:
        type: text
    aggregate:
      balance:
        from: amount
        fn: sum
      transfer_count:
        from: id
        fn: count

Supported aggregation functions: sum, count, avg, min, max

Multi-Sink Considerations

• Each sink reads from a from: field — different sinks can read from different transforms
• Sinks are independent — one failing doesn't block others
• Use different batch_size / batch_flush_interval per sink based on latency needs
• ClickHouse supports parallelism: N for concurrent writers (default 1)

Webhook Sinks Without Secrets

Webhooks can use a direct URL instead of a secret when no auth headers are needed:

sinks:
  my_webhook:
    type: webhook
    from: my_transform
    url: https://my-lambda.us-west-2.on.aws/

---

Pipeline Splitting Decisions

One Pipeline vs. Multiple

Use one pipeline when:

• All data comes from the same source
• Transforms share intermediate results (e.g., a shared decode step)
• You want atomic deployment of the whole flow

Split into multiple pipelines when:

• Different data sources with no shared transforms
• Different lifecycle needs (one is stable, another changes frequently)
• Different resource requirements (one needs l, another needs s)
• Different chains with independent processing (templated pattern)

Keeping Pipelines Focused

A pipeline should ideally do one logical thing:

| Pipeline | Focus | | ------------------------------- | ----------------------------------- | | dex-trades | Trade events → Postgres | | dex-activities | All activity types → ClickHouse DWH | | token-balances | Token balances → Postgres | | base-balance-streaming | Base balances → ClickHouse + webhook |

Even though trades are a subset of activities, they're separate pipelines because they serve different consumers (application DB vs data warehouse).

---

Streaming vs Job Mode

Turbo pipelines have two execution modes:

Streaming Mode (Default)

name: my-streaming-pipeline
resource_size: s
# job: false  (default — omit this field)

• Runs continuously, processing data as it arrives
• Maintains checkpoints for exactly-once processing
• Use for real-time feeds, dashboards, APIs

Job Mode (One-Time Batch)

name: my-backfill-job
resource_size: l
job: true

• Runs to completion and stops automatically
• Auto-deletes resources ~1 hour after completion
• Must delete before redeploying — cannot update a job pipeline, must goldsky turbo delete first
• Cannot use restart — use delete + apply instead
• Use for historical backfills, one-time data migrations, snapshot exports

When to Use Which

| Scenario | Mode | Why | | ------------------------------------ | ----------- | ---------------------------------------------- | | Real-time dashboard | Streaming | Continuous updates needed | | Backfill 6 months of history | Job | One-time, stops when done | | Real-time + catch-up on deploy | Streaming | start_at: earliest does backfill then streams| | Export data to S3 once | Job | No need for continuous processing | | Webhook notifications on events | Streaming | Needs to react as events happen | | Load test with historical data | Job | Process and inspect, then discard |

Job Mode with Bounded Ranges

Combine job mode with start_at: earliest and an end_block to process a specific range:

name: historical-export
resource_size: l
job: true

sources:
  logs:
    type: dataset
    dataset_name: ethereum.raw_logs
    version: 1.0.0
    start_at: earliest
    end_block: 19000000
    filter: >-
      address = '0xdac17f958d2ee523a2206206994597c13d831ec7'

---

Dynamic Table Architecture

Dynamic tables enable runtime-updatable lookup data within a pipeline. They're the Turbo answer to the "no joins in streaming SQL" limitation.

Pattern: Dynamic Allowlist/Blocklist

                    ┌──────────────────────┐
                    │  External Updates     │
                    │  (Postgres / REST)    │
                    └──────────┬───────────┘
                               ▼
source ──→ sql transform ──→ [dynamic_table_check()] ──→ sink

The SQL transform filters records against the dynamic table. The table contents can be updated externally without pipeline restart.

Pattern: Lookup Enrichment

source ──→ decode ──→ filter ──→ sql (with dynamic_table_check) ──→ sink
                                        ▲
                              [token_metadata table]
                              (postgres-backed)

Store metadata (token symbols, decimals, protocol names) in a PostgreSQL table. Reference it in transforms for enrichment.

Backend Decisions

| Backend | backend_type | When to Use | | ----------- | -------------- | ----------------------------------------------------------------- | | PostgreSQL | Postgres | Data managed by external systems, shared across pipeline restarts | | In-memory | InMemory | Auto-populated from pipeline data, ephemeral, fastest lookups |

Sizing Considerations

• Dynamic tables add memory overhead proportional to table size
• For large lookup tables (>100K rows), use Postgres backend
• For small, frequently-changing lists (<10K rows), InMemory is faster
• Dynamic table queries are async — they add slight latency per record

For full dynamic table configuration syntax and examples, see /turbo-transforms.

---

Related

• /turbo-builder — Build and deploy pipelines interactively using these architecture patterns
• /turbo-doctor — Diagnose and fix pipeline issues
• /turbo-pipelines — Pipeline YAML configuration reference
• /turbo-transforms — SQL, TypeScript, and dynamic table transform reference
• /datasets — Blockchain dataset and chain prefix reference
• /secrets — Sink credential management
• /turbo-monitor-debug — Monitoring and debugging reference
• /turbo-lifecycle — Pipeline lifecycle command reference