Skip to main content

Snowflake

Overview​

Snowflake is a cloud-native data warehouse. SaddleData connects to Snowflake as a Destination, allowing you to sync data from various sources into your Snowflake data warehouse.

Prerequisites​

To connect to Snowflake, you will need:

  • A Snowflake account.
  • A user with permissions to create tables and write data in your target schema.
  • A Warehouse for compute resources.

Configuration​

When creating a Snowflake Integration, you will need to provide the following information:

  • Account: Your Snowflake account identifier (e.g., abcdefg-foc12345.snowflakecomputing.com or orgname-accountname).
  • User: The username to connect with.
  • Password: The password for the user.
  • Database: The name of the destination database.
  • Schema: The name of the destination schema.
  • Warehouse: The name of the warehouse to use for loading data.
  • Role: The role to assume (e.g., ACCOUNTADMIN or a custom role with specific privileges).

Sync Modes​

When using Snowflake as a destination, you can choose from the following sync modes:

  • Full Refresh - Overwrite: Replaces all data in the destination table.
  • Incremental - Append: Appends new records to the destination table.
  • Incremental - Deduped (Upsert): Updates existing rows and inserts new rows based on a primary key using the MERGE command.

Schema Evolution​

SaddleData supports schema drift handling for Snowflake:

  • Automatic Migration: If enabled, SaddleData will automatically create the destination table if it doesn't exist and add new columns as they appear in the source data (ALTER TABLE).
  • Lazy Initialization: The table and schema are adapted dynamically based on the first batch of data received in each run.

Declarative Configuration​

apiVersion: v1
kind: Connection
metadata:
  name: snowflake-connection
spec:
  connectorId: snowflake
  configuration:
    account: my-account
    user: saddledata
    password: '********'
    database: my_database
    schema: my-schema
    warehouse: my-warehouse
    role: my-role
    capability: destination