Skip to main content

Flows

A Flow is the core component of Saddle Data. It's a pipeline that moves data from a Source Connection to a Destination Connection.

Each Flow is defined by a configuration that specifies:

  • The source connection.
  • The destination connection.
  • The schema mapping (tables, columns, and types).
  • An optional series of in-flight transformations.

A Flow can be run manually at any time, scheduled to run automatically via cron, or orchestrated to run as part of a chained pipeline.

The Schema Tab​

The Schema Tab in the Flow Editor is where you define exactly what data moves and how it should look at the destination. It features a side-by-side view:

  • Source Schema (Left): Browse and select the tables and columns available in your source connection.
  • Field Mapping & Types (Right): Configure the destination for each selected source table.

Field Mapping & Renaming​

You can customize the destination table names and column names. This is particularly useful for flattening structures or aligning with existing warehouse naming conventions.

Smart Casting & Types​

Saddle Data uses a standardized set of Saddle Types (String, Integer, Float, Boolean, DateTime, JSON, and Binary) to ensure reliable data movement.

  • Implicit Casting: When you select a destination type that differs from the source, the worker automatically attempts to cast the data in-flight (e.g., converting a numeric string from a spreadsheet into a true Integer for a database).
  • Targeted Creation: If the destination table doesn't exist, Saddle Data uses these type mappings to generate the correct CREATE TABLE statement.

Sync Modes​

Sync modes are configured per-table mapping in the Schema tab:

  • Full Refresh - Overwrite: This mode replaces all data in the destination table with the data from the source. It's a destructive operation, so be careful when using it.

  • Incremental - Append: This mode only syncs new records from the source to the destination. It's a non-destructive operation. This mode is ideal for append-only data streams.

  • Incremental - Deduped (Upsert): This mode syncs new records and updates existing records in the destination. This requires a primary key to be defined for the table. It's a non-destructive operation and is the recommended mode for most incremental syncs.

It's important to note that not all destinations support all sync modes. The UI will guide you and provide warnings if there's a mismatch between the chosen sync mode and the destination's capabilities.

Schema Drift Handling​

Source schemas change over time—columns are added, removed, or renamed. Saddle Data allows you to define a Schema Change Policy for each table mapping in the Schema tab to control how these changes are handled.

Policies​

You can select one of the following policies for each mapping:

  • Automatically Update Destination (allow_all):

    • Behavior: When a new column is detected in the source, Saddle Data automatically adds the corresponding column to the destination table and continues the sync without interruption.
    • Use Case: Best for development environments or pipelines where the destination schema should strictly mirror the source.

  • Pause Pipeline (block_on_new_column):

    • Behavior: When a new or missing column is detected, the flow run is immediately halted with a status of drift_detected. A notification is sent (if configured).
    • Resolution: You must review the drift in the Flow Details page. You can then choose to "Approve & Sync", which will perform a one-time schema migration to align the destination with the source and resume the pipeline.
    • Use Case: Essential for production environments where schema changes must be audited or could break downstream dependencies.

Drift Resolution Workflow​

If a flow is paused due to schema drift:

  1. Navigate to the Flow Details page.
  2. Look for the run with the drift_detected status (indicated by an orange warning icon).
  3. Click View Drift to see the specific changes (e.g., "New columns: user_score, last_login").
  4. Click Approve & Sync. Saddle Data will update your flow configuration to include the new columns and trigger a run to apply the changes to the destination.

Governance & Security​

Flows in Saddle Data are inherently "governance-aware." If you are using the Data Catalog or have defined Data Governance policies, your flows will automatically reflect these security requirements.

  • Automatic Enforcement: If a column in your source mapping is tagged with a security label (like PII), the required transformation (e.g., HASH) will be automatically injected into your flow.
  • Managed Locks: These security-driven transformations are "managed." They appear in the Flow Editor with a lock icon and cannot be modified or removed by standard users, ensuring continuous compliance.
  • Circuit Breakers: At runtime, the Worker verifies that all active governance policies are being met. If a policy is bypassed, the run is immediately aborted to prevent a data leak.