Snowflake 目的地
Replicate Supabase Postgres changes to Snowflake.
公开测试版
Supabase Pipelines 目前处于公开测试阶段。随着我们继续开发产品,功能和表现可能会有所变化。
🌐 Supabase Pipelines is currently in public alpha. Features and behavior may change as we continue developing the product.
Snowflake 是一个托管的数据平台。Supabase Pipelines 会将每个 Postgres 表复制到 Snowflake,作为仅追加的更改历史记录。
将数据复制到 Snowflake:
🌐 To replicate data to Snowflake:
源表要求 #
🌐 Source table requirements
Postgres 发布中启用的操作决定了所需的 REPLICA IDENTITY:
🌐 The operations enabled in the Postgres publication determine the required REPLICA IDENTITY:
| 已发布操作 | 所需副本标识 |
|---|---|
INSERT 仅 | 不需要行标识。 |
DELETE | 主键、标识索引 (USING INDEX) 或完整标识 (FULL)。发布所有标识列。 |
UPDATE | REPLICA IDENTITY FULL。 |
在发布更新之前设置完整副本身份:
🌐 Set full replica identity before publishing updates:
alter table public.your_table replica identity full;REPLICA IDENTITY FULL 会增加 WAL 的体积,但允许管道在 Postgres 忽略未更改的外部 TOAST 值时构建完整的新行。它只适用于新的 WAL 记录。如果保留的 WAL 已经包含不兼容的更新,更改设置后需要为受影响的表重启复制。
准备 Snowflake 资源 #
🌐 Prepare Snowflake resources
在你创建管道之前,先准备一个专用的 Snowflake 数据库和一个用于复制表的空模式。管道还需要一个可以创建和管理这些表的服务用户和角色。服务用户和角色使用不带引号的标识符,这样 Snowflake 会将它们存储为大写。管道在认证时也会把账号和用户名转换为大写。
🌐 Before you create a pipeline, prepare a dedicated Snowflake database and an empty schema for the replicated tables. Pipelines also needs a service user and role that can create and manage those tables. Use unquoted identifiers for the service user and role so Snowflake stores them in uppercase. Pipelines also converts the account and user names to uppercase during authentication.
以 Snowflake 管理员身份运行此设置,根据需要更改示例名称:
🌐 Run this setup as a Snowflake administrator, changing the example names as needed:
create role if not exists PIPELINES_ROLE;create user if not exists PIPELINES_USER type = service;grant role PIPELINES_ROLE to user PIPELINES_USER;alter user PIPELINES_USER set default_role = PIPELINES_ROLE;create database if not exists PIPELINES_DB;create schema if not exists PIPELINES_DB.REPLICATED;grant usage on database PIPELINES_DB to role PIPELINES_ROLE;grant usage on schema PIPELINES_DB.REPLICATED to role PIPELINES_ROLE;grant create table on schema PIPELINES_DB.REPLICATED to role PIPELINES_ROLE;在这个例子中,Pipelines 使用 PIPELINES_ROLE 创建目标表。无论你选择什么名字,角色必须保留所有权,这样 Pipelines 在需要时才能修改、截断或删除表。不要提前在其他角色下创建表。
🌐 In this example, Pipelines creates the destination tables using PIPELINES_ROLE. Whatever name you choose, the role must retain ownership so Pipelines can alter, truncate, or drop the tables when required. Do not create the tables in advance under another role.
Snowflake 会为每个表自动创建一个名为 <TABLE>-STREAMING 的托管默认管道。你不需要提供虚拟仓库、阶段或管道来进行数据摄取。有关所需权限,请参见 Snowpipe 流式访问权限。
🌐 Snowflake automatically creates a managed default pipe named <TABLE>-STREAMING for each table. You don't need to provide a virtual warehouse, stage, or pipe for ingestion. See Snowpipe Streaming access privileges for the required permissions.
为了让摄取资源与下游工作负载分开,查询和转换使用不同的角色和仓库。
🌐 To keep ingestion resources separate from downstream workloads, use a different role and warehouse for queries and transformations.
保持 SQL 和流处理角色一致 #
🌐 Keep the SQL and streaming roles aligned
管道通过不同的接口连接到 Snowflake,用于 SQL 请求和流处理。这两个接口都需要使用相同的角色:
🌐 Pipelines connects to Snowflake through separate interfaces for SQL requests and streaming. Both interfaces need to use the same role:
- SQL 请求在仪表板的 高级设置 下可以使用可选的 角色。当 角色 为空时,它们会使用用户的默认角色。
- Snowpipe Streaming 使用用户的
DEFAULT_ROLE。它不使用可选的 Role 设置。
将上面的权限授予一个专用角色,例如 PIPELINES_ROLE,然后将其设置为服务用户的 DEFAULT_ROLE。在仪表板中,要么留空 Role,要么明确输入相同的角色。这样可以保持 SQL 验证和流处理的一致性。
🌐 Grant the permissions above to a dedicated role such as PIPELINES_ROLE, then set it as the service user's DEFAULT_ROLE. In the Dashboard, either leave Role empty or enter the same role explicitly. This keeps SQL validation and streaming aligned.
生成一对密钥 #
🌐 Generate a key pair
Pipelines 使用 RSA 密钥对来对 Snowflake 进行身份验证。密钥至少要 2048 位,Snowflake 推荐使用 PKCS #8 格式。在你想存放密钥的目录里运行下面的命令之一。命令会在该目录下创建一个名为 rsa_key.p8 的私钥。
🌐 Pipelines uses an RSA key pair to authenticate with Snowflake. The key must be at least 2048 bits, and Snowflake recommends the PKCS #8 format. Run one of the following commands from the directory where you want to store the key. The command creates a private key named rsa_key.p8 in that directory.
对于未加密的私钥:
🌐 For an unencrypted private key:
openssl genrsa 2048 | openssl pkcs8 -topk8 \ -inform PEM -out rsa_key.p8 -nocrypt或者,如果是用密码保护的私钥:
🌐 Or, for a passphrase-protected private key:
openssl genrsa 2048 | openssl pkcs8 -topk8 -v2 des3 \ -inform PEM -out rsa_key.p8从私钥创建公钥。Snowflake 使用公钥来验证用 rsa_key.p8 签名的连接:
🌐 Create a public key from the private key. Snowflake uses the public key to verify connections signed with rsa_key.p8:
openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub打开 rsa_key.pub,复制 BEGIN PUBLIC KEY 和 END PUBLIC KEY 行之间的文本。在下面的语句中,用复制的文本替换 <public-key-body> 占位符:
🌐 Open rsa_key.pub and copy the text between the BEGIN PUBLIC KEY and END PUBLIC KEY lines. In the following statement, replace the <public-key-body> placeholder with the copied text:
alter user PIPELINES_USER set rsa_public_key = '<public-key-body>';在仪表板配置目标时,将完整的 rsa_key.p8 文件粘贴或上传到 私钥。保留其原始的 PEM 头部和尾部。如果密钥有密码保护,请将密码输入到 私钥密码。
🌐 When you configure the destination in the Dashboard, paste or upload the complete rsa_key.p8 file into Private key. Preserve its original PEM header and footer. If the key is passphrase-protected, enter the passphrase into Private key passphrase.
保密 rsa_key.p8 及其密码。仪表板接受未加密的 PKCS #1 和 PKCS #8 密钥。它也接受带密码的加密 PKCS #8 密钥,但不接受加密的 PKCS #1 密钥。
🌐 Keep rsa_key.p8 and its passphrase secret. The Dashboard accepts unencrypted PKCS #1 and PKCS #8 keys. It also accepts encrypted PKCS #8 keys with a passphrase, but not encrypted PKCS #1 keys.
查看 Snowflake 密钥对认证 来验证公钥指纹,并使用 RSA_PUBLIC_KEY_2 轮换密钥。
🌐 See Snowflake key-pair authentication to verify the public-key fingerprint and rotate keys with RSA_PUBLIC_KEY_2.
找到账户标识 #
🌐 Find the account identifier
在 Snowflake 中运行此查询:
🌐 Run this query in Snowflake:
select current_organization_name() || '-' || current_account_name();将结果用作账户 ID,例如 MYORG-MYACCOUNT。该字段也接受旧版的一部分账户定位符,但不接受完整的 URL 或点分定位符和区域主机名。账户 ID 最多可以包含 63 个字符。详情请参见 Snowflake 账户标识符。
🌐 Use the result as the Account ID, for example MYORG-MYACCOUNT. The field also accepts legacy one-part account locators, but not full URLs or dotted locator-and-region hostnames. Account IDs can contain up to 63 characters. See Snowflake account identifiers for details.
将 Snowflake 配置为目标 #
🌐 Configure Snowflake as a destination
按照 设置管道 中的步骤操作。当系统提示选择目标时,选择 Snowflake 并输入以下设置:
🌐 Follow the steps in Set up Pipelines. When prompted to choose a destination, select Snowflake and enter the following settings:
| 字段 | 值 |
|---|---|
| 账户 ID | MYORG-MYACCOUNT,例如;使用组织账户标识符 |
| 用户 | PIPELINES_USER,或者你的未加引号的服务用户 |
| 数据库 | PIPELINES_DB,或者你的目标数据库 |
| 模式 | REPLICATED,或者你专用的目标模式 |
| 角色 | 留空以使用服务用户的默认角色,或者显式输入相同的角色 |
| 私钥 | 粘贴或上传完整的 rsa_key.p8 私钥 PEM 文件 |
| 私钥密码短语 | 仅针对加密的 PKCS #8 密钥 |
请输入数据库和架构标识符,确保与 Snowflake 中存储的完全一致。未加引号的标识符会以大写存储。选择一个靠近托管管道区域的账户。
🌐 Enter the database and schema identifiers exactly as stored in Snowflake. Unquoted identifiers are stored in uppercase. Choose an account near the managed pipeline region.
点击 开始管道,然后完成验证和费用确认。
🌐 Click Start pipeline, then complete the validation and cost confirmations.
它是怎么运作的 #
🌐 How it works
Pipelines 使用 Snowflake 的 SQL REST API 来验证数据库和架构。它还使用该 API 来创建、更新和重新创建目标表,并应用源 TRUNCATE 操作。Pipelines 通过 Snowpipe Streaming 发送初始和持续的行数据。
🌐 Pipelines uses Snowflake's SQL REST API to validate the database and schema. It also uses the API to create, update, and recreate destination tables and to apply source TRUNCATE operations. Pipelines sends initial and ongoing row data through Snowpipe Streaming.
验证会检查身份验证、数据库和架构的可见性,以及 QUOTED_IDENTIFIERS_IGNORE_CASE 是否为 FALSE。它不会验证角色是否可以创建或拥有表,或通过 Snowpipe Streaming 写入数据。
🌐 Validation checks authentication, database and schema visibility, and that QUOTED_IDENTIFIERS_IGNORE_CASE is FALSE. It does not verify that the role can create or own tables or write through Snowpipe Streaming.
目标表名 #
🌐 Destination table names
Pipelines 会将每个 Postgres 的 schema 和表名组合成一个 Snowflake 表名。已有的下划线会加倍,两个名字用一个下划线连接,然后结果会被转换成大写:
🌐 Pipelines combines each Postgres schema and table name into one Snowflake table name. Existing underscores are doubled, the two names are joined with a single underscore, and the result is converted to uppercase:
| Postgres 表 | Snowflake 表 |
|---|---|
public.orders | PUBLIC_ORDERS |
sales_eu.order_items | SALES__EU_ORDER__ITEMS |
Postgres 的模式和表名不能以 _ 开头或结尾,也不能包含 " 或 ;。仅大小写不同的名字会映射到相同的 Snowflake 名称。为了避免冲突,请使用小写的 Postgres 名称。源列名会作为带引号的标识符保留,除了 保留的元数据名称。
🌐 Postgres schema and table names cannot start or end with _ or contain " or ;. Names that differ only in case map to the same Snowflake name. Use lowercase Postgres names to avoid collisions. Source column names are preserved as quoted identifiers, except for the reserved metadata names.
仅追加的变更历史 #
🌐 Append-only change history
每个目标表都包含复制的源列以及两个 VARCHAR NOT NULL 元数据列:
🌐 Each destination table contains the replicated source columns plus two VARCHAR NOT NULL metadata columns:
| 列 | 含义 |
|---|---|
_cdc_operation | 小写操作:insert、update 或 delete。 |
_cdc_sequence_number | 固定宽度的十六进制提交 LSN 和事务序号,例如 00000000016b3740/0000000000000002。 |
元数据名称是保留的,源列不能使用。初始同步行使用 insert,共享序列号使用 0000000000000000/0000000000000000。
🌐 The metadata names are reserved and can't be used by source columns. Initial-sync rows use insert and the shared sequence number 0000000000000000/0000000000000000.
Snowflake 表包含事件历史,而不是当前状态的副本:
🌐 Snowflake tables contain an event history rather than a current-state replica:
- 插入会添加新行。
- 更新会添加完整的新行。它不会添加之前的图片。
- 删除操作会附加完整旧行数据给
REPLICA IDENTITY FULL。对于主键或USING INDEX身份列,它只会发送身份列。其他列可以包含目标默认值或NULL;不要把它们当作被删除行的原始值。 - 截断源表也会截断 Snowflake 表并重置其流状态。它不会附加截断事件。
序列号会改变顺序,但不是全局唯一的事件ID。Snowpipe 提交的偏移量可以抑制例行重放,但消费者仍然必须容忍重复处理。
🌐 The sequence number orders changes but is not a globally unique event ID. Snowpipe committed offsets suppress routine replay, but consumers must still tolerate duplicate processing.
【重启表】(/docs/guides/database/replication/pipelines-monitoring#restarting-tables) 会删除 Snowflake 表及其管理的流状态,这会清除复制的历史记录。重启无法恢复过去的事件。相比之下,【从发布中移除表】(/docs/guides/database/replication/pipelines#removing-tables-from-replication) 会保留其目标历史记录。
查询复制的数据 #
🌐 Query replicated data [#query-and-materialize-current-state]
使用复制的变更历史来构建用于报表和分析的当前状态数据集。Pipelines 维护历史表,而你来维护查询、视图或从中读取的动态表。
🌐 Use the replicated change history to build current-state datasets for reporting and analytics. Pipelines maintains the history table, while you maintain the queries, views, or dynamic tables that read from it.
在你开始之前 #
🌐 Before you start
示例使用了一个名为 public.orders 的源表,包含列 id 和 status。数据管道会将它复制到 PIPELINES_DB.REPLICATED.PUBLIC_ORDERS。请将这些名称替换为你自己的,并在初始同步完成后再把结果当作完整的副本。
🌐 The examples use a source table named public.orders with the columns id and status. Pipelines replicates it to PIPELINES_DB.REPLICATED.PUBLIC_ORDERS. Replace these names with your own, and wait for the initial sync to finish before treating the results as a complete replica.
选择一个独特的、非空的标识,当行更新时不会改变。例子里用的是 id。如果你使用复合键,在 partition by 中要包含每个键列,比如 partition by "tenant_id", "id"。发布和删除事件必须包含相同的列。单靠 REPLICA IDENTITY FULL 无法保证行的唯一性。
🌐 Choose a unique, non-null identity that does not change when a row is updated. The examples use id. If you use a composite key, include every key column in partition by, such as partition by "tenant_id", "id". The publication and delete events must include the same columns. REPLICA IDENTITY FULL alone does not make rows unique.
如果身份值发生变化,旧的身份可能仍会出现在结果中。Pipelines 会在添加更新后的行之前,不会先删除之前的身份。对这种模式使用不可变的键。
🌐 If an identity value changes, the old identity can remain in the results. Pipelines appends the updated row without first appending a delete for the previous identity. Use an immutable key for this pattern.
在 Pipelines 管理的 REPLICATED 模式之外的模式中创建派生对象,使用单独的分析角色和仓库。示例使用了 ANALYTICS_ROLE、ANALYTICS_WH 和 PIPELINES_DB.ANALYTICS。请向你的 Snowflake 管理员请求准备这些资源,并授予分析角色:
🌐 Create derived objects in a schema outside the Pipelines-managed REPLICATED schema, using a separate analytics role and warehouse. The examples use ANALYTICS_ROLE, ANALYTICS_WH, and PIPELINES_DB.ANALYTICS. Ask your Snowflake administrator to prepare these resources and grant the analytics role:
USAGE在仓库、数据库以及两个模式上。SELECT在复制的表上。- 在分析架构上使用
CREATE VIEW来创建视图,或者使用CREATE DYNAMIC TABLE来创建动态表。
这个角色必须对运行示例的 Snowflake 用户可用。保持被复制表的所有权在 PIPELINES_ROLE。有关完整的权限要求,请参阅 Snowflake 的动态表访问控制。
🌐 The role must be available to the Snowflake user running the examples. Keep ownership of the replicated table with PIPELINES_ROLE. See Snowflake's dynamic table access control for the full privilege requirements.
查询当前状态 #
🌐 Query current state
用你的分析角色在 Snowflake SQL 工作表中运行这些语句:
🌐 Run these statements in a Snowflake SQL worksheet with your analytics role:
use role ANALYTICS_ROLE;use warehouse ANALYTICS_WH;select "id", "status"from PIPELINES_DB.REPLICATED.PUBLIC_ORDERSqualify row_number() over ( partition by "id" order by "_cdc_sequence_number" desc) = 1and "_cdc_operation" != 'delete';查询会为每个最新操作不是 delete 的身份返回一行。固定宽度的序列字符串决定了哪个变化是最新的,重复的相同事件会合并成一行结果。源列和元数据列名要保留双引号,因为 Pipelines 会把它们创建为区分大小写的标识符。
🌐 The query returns one row for each identity whose latest operation is not delete. The fixed-width sequence string determines which change is the latest, and repeated copies of the same event collapse into one result row. Keep the double quotes around source and metadata column names because Pipelines creates them as case-sensitive identifiers.
把删除条件保留在 qualify 里。把它移到 where "_cdc_operation" != 'delete' 会在行被排名之前就删除事件,这可能会恢复被删除行的旧版本。Snowflake 的 QUALIFY 参考 解释了这个评估顺序。
🌐 Keep the delete condition inside qualify. Moving it to where "_cdc_operation" != 'delete' would remove delete events before the rows are ranked, which could bring back an older version of a deleted row. Snowflake's QUALIFY reference explains this evaluation order.
要重用分析工具中的查询,可以将其保存为视图:
🌐 To reuse the query from an analytics tool, save it as a view:
create view PIPELINES_DB.ANALYTICS.ORDERS_CURRENT_VIEW asselect "id", "status"from PIPELINES_DB.REPLICATED.PUBLIC_ORDERSqualify row_number() over ( partition by "id" order by "_cdc_sequence_number" desc) = 1and "_cdc_operation" != 'delete';普通视图存储的是查询定义,而不是结果的单独副本。每次读取都会根据查询时可用的历史记录得出当前状态。请参阅 Snowflake 的视图与动态表比较。
🌐 A regular view stores the query definition rather than a separate copy of the results. Each read derives the current state from the history available at query time. See Snowflake's comparison of views and dynamic tables.
用动态表格实现 #
🌐 Materialize with a dynamic table
动态表会存储查询结果,并在复制的历史记录发生变化时刷新它们。当你需要一个始终保持最新状态的数据集,而又不想维护定时合并任务时,可以使用它。
🌐 A dynamic table stores the query results and refreshes them as the replicated history changes. Use one when you need a maintained current-state dataset without maintaining a scheduled merge task.
-
请让复制表的所有者在 Snowflake 中启用变更跟踪。这是表的设置,不是对复制列或数据的更改。以
PIPELINES_ROLE或其他继承所有权的角色运行:alter table PIPELINES_DB.REPLICATED.PUBLIC_ORDERSset change_tracking = true;分析角色不拥有复制的表,因此在创建动态表时无法自动启用变更跟踪。请参阅 Snowflake 的变更跟踪要求。
-
切换到分析角色并创建动态表:
use role ANALYTICS_ROLE;use warehouse ANALYTICS_WH;create dynamic table PIPELINES_DB.ANALYTICS.ORDERS_CURRENTtarget_lag = '5 minutes'warehouse = ANALYTICS_WHrefresh_mode = incrementalinitialize = on_createasselect "id", "status"from PIPELINES_DB.REPLICATED.PUBLIC_ORDERSqualify row_number() over (partition by "id" order by "_cdc_sequence_number" desc) = 1and "_cdc_operation" != 'delete';initialize = on_create会在创建完成之前填充动态表。显式的refresh_mode = incremental如果你的自定义查询无法增量刷新,会让创建失败,而不是通过AUTO选择完全刷新。查看 Snowflake 的 刷新模式 和CREATE DYNAMIC TABLE参考。 -
检查刷新模式并读取已物化的行:
show dynamic tables like 'ORDERS_CURRENT'in schema PIPELINES_DB.ANALYTICS;select "id", "status"from PIPELINES_DB.ANALYTICS.ORDERS_CURRENT;确认
refresh_mode是INCREMENTAL,并且调度正在运行。使用 Snowflake 的刷新监控 检查最后一次成功刷新和任何错误。在插入、更新或删除操作到达复制表后,下次成功刷新会在ORDERS_CURRENT中体现。
五分钟的 target_lag 是一个相对于 Snowflake 已有历史数据的示例新鲜度目标。它既不是固定的刷新计划,也不是来自 Postgres 的端到端延迟保证。管道复制延迟和动态表刷新延迟都会影响新鲜度。详情请参见 Snowflake 的 目标延迟指南。
🌐 The five-minute target_lag is an example freshness target relative to the history already in Snowflake. It is neither a fixed refresh schedule nor an end-to-end latency guarantee from Postgres. Both pipeline replication lag and dynamic-table refresh lag affect freshness. See Snowflake's target lag guide.
刷新动态表会消耗仓库计算资源,而它的物化结果会占用存储空间。这些成本是额外的,除了数据摄取和查询之外。先从满足你报告需求的新鲜度目标开始,然后用代表性的工作负载测试其成本和刷新行为。专用仓库可以帮助隔离刷新成本。参见 Snowflake 的动态表成本指南。
🌐 Refreshing a dynamic table consumes warehouse compute, while its materialized results consume storage. These costs are additional to ingestion and querying. Start with a freshness target that meets your reporting needs, then test its cost and refresh behavior with a representative workload. A dedicated warehouse can help isolate refresh costs. See Snowflake's dynamic table cost guide.
使用流和任务 #
🌐 Use streams and tasks
Snowflake 的 streams 和 tasks 可以通过定期的 MERGE 语句维护一个独立的表。在你需要对更新过程或时间表进行更多控制时可以使用它们。Snowflake 的 SCD 类型 1 示例 将这种方法与动态表进行了比较。
🌐 Snowflake streams and tasks can maintain a separate table through scheduled MERGE statements. Use them when you need more control over the update procedure or schedule. Snowflake's SCD Type 1 examples compare this approach with dynamic tables.
将合并操作适配到 Pipelines 创建的 "_cdc_operation" 和 "_cdc_sequence_number" 列。历史表上的数据流会看到每一行追加的记录,包括表示源更新和删除的行。你的工作需要解读这些操作,加载现有历史,容忍重放,并在源截断或管道表重置后重建当前状态。
🌐 Adapt the merge to the "_cdc_operation" and "_cdc_sequence_number" columns created by Pipelines. A stream on the history table sees every appended row, including rows that represent source updates and deletes. Your job must interpret those operations, load the existing history, tolerate replay, and rebuild the current state after a source truncate or pipeline table reset.
维护派生对象 #
🌐 Maintain derived objects
Pipelines 会维护复制的历史表,但不会更新你的视图或动态表定义。
🌐 Pipelines maintains the replicated history table, but does not update your view or dynamic-table definitions.
| 变更 | 需要做的事情 |
|---|---|
来源 TRUNCATE | 直接查询或视图会读取截断的历史记录。在依赖其内容之前,检查动态表是否完成刷新。 |
| 流水线表重置 | 等待新的初始同步。对重新创建的历史表重新应用表特定的读取权限和变更跟踪。检查依赖对象,如果动态表无法刷新则重新创建。 |
| 添加、重命名或删除源列 | 查看显式列列表。必要时将新列添加到你的定义中。更新或重新创建引用已重命名或删除列的派生对象。 |
重新创建动态表会重新初始化其内容并使用计算功能。有关需要重新初始化的更改,请参阅 Snowflake 的动态表修改指南。
🌐 Recreating a dynamic table initializes its contents again and uses compute. See Snowflake's dynamic table modification guide for changes that require reinitialization.
类型映射 #
🌐 Type mapping
管道会用这些映射创建 Snowflake 列:
🌐 Pipelines creates Snowflake columns with these mappings:
| Postgres 类型 | Snowflake 类型 |
|---|---|
boolean | BOOLEAN |
smallint、integer、bigint | SMALLINT、INTEGER、BIGINT |
real、double precision | FLOAT、DOUBLE |
date、time | DATE、TIME |
timestamp、timestamp with time zone | TIMESTAMP_NTZ、TIMESTAMP_TZ |
json、jsonb | VARIANT |
| 一维数组 | ARRAY |
oid | BIGINT |
| 其他类型 | VARCHAR |
管道会将字符和文本类型、numeric、time with time zone、interval、uuid、bytea、位字符串以及自定义或未知类型映射到 VARCHAR。它会序列化这些值,而不是以本地 Snowflake 类型存储它们。对于 bytea,序列化后的值是小写的十六进制字符串。
🌐 Pipelines maps character and text types, numeric, time with time zone, interval, uuid, bytea, bit strings, and custom or unknown types to VARCHAR. It serializes these values instead of storing them as native Snowflake types. For bytea, the serialized value is a lowercase hexadecimal string.
还有额外限制:
🌐 Additional limits apply:
- 不支持多维数组。一维数组的非默认下界不会被保留。
- 非有限的
real和double precision值会被拒绝。非有限的numeric值会以字符串形式保存在VARCHAR列中。 - 未压缩的序列化行大于 2 MiB 会被拒绝。
- 源主键、唯一性、检查、长度、精度和可空性约束不会被复制。只有两个 CDC 元数据列是
NOT NULL。
支持模式更改 #
🌐 Schema change support
管道支持:
🌐 Pipelines supports:
- 添加、重命名或删除列
- 在跟踪的表上添加或删除已发布的列
在 Snowflake 中,复制的列仍然可以为空,并且对现有列默认值的更改不会被传播。当表第一次创建时,Pipelines 可以复制兼容的字面默认值。它还可以为后来在 Postgres 中添加的列复制字符串、数字或布尔字面默认值。其他默认值会被省略,尽管 Postgres 仍会通过复制提供源值。
🌐 Replicated columns remain nullable in Snowflake, and changes to existing column defaults are not propagated. When a table is first created, Pipelines can copy compatible literal defaults. It can also copy string, numeric, or boolean literal defaults for columns added later in Postgres. Other defaults are omitted, although Postgres still supplies the source values through replication.
模式更改也会影响存储的历史记录。重命名列会改变早期事件中的列名,删除列会移除它的历史值,而添加带默认值的列可以填充旧的行。
🌐 Schema changes also affect the stored history. Renaming a column changes its name in earlier events, dropping a column removes its historical values, and adding a column with a default can populate older rows.
之前被排除的列会在没有默认值的情况下被添加回来,这些列的历史事件会保留为 NULL。删除已发布的列会丢失其目标值;再次添加时不会恢复这些值。
🌐 Previously excluded columns are added without defaults, leaving historical events NULL for those columns. Removing a published column drops its destination values; adding it again does not restore them.
有关类型更改、不受支持的更改以及中断的模式更改,请参阅共享的 模式更改行为和恢复。除了 启用更改跟踪 外,请勿手动更改托管目标对象。
🌐 For type changes, unsupported changes, and interrupted schema changes, see the shared schema-change behavior and recovery. Apart from enabling change tracking, do not alter managed destination objects manually.
故障排除 #
🌐 Troubleshooting
| 症状 | 检查内容 |
|---|---|
| 认证失败 | 账户标识符、用户、密钥指纹和 PEM 以及密码短语。对于未加密的密钥可省略密码短语。 |
| 未找到数据库或模式 | 确认精确的标识符大小写以及两个对象上的 USAGE 权限 |
| 验证通过但表创建失败 | 权限和所有权,包括 CREATE TABLE |
| 表已创建,但写入失败 | SQL 和流式角色对齐、表权限,以及对账户控制和发现的 Snowpipe 吞吐端点的网络访问 |
| 更新或删除失败 | 源副本标识和发布的列 |
| 一行被拒绝 | 类型和行大小限制 和 保留的元数据列 |
| 架构更改失败 | 支持的更改;不要手动修复托管表 |
使用 管道监控 来检查错误。如有未解决的失败,请携带管道 ID 和错误详情 联系支持。
🌐 Use pipeline monitoring to inspect errors. For unresolved failures, contact support with the pipeline ID and error details.
额外资源 #
🌐 Additional resources