编程 ClickHouse 26.9 的两个实用改动:LIMIT 支持 AFTER/UNTIL 条件边界,物化视图可 APPEND INCREMENTAL 增量刷新

2026-09-26 20:01:52

ClickHouse 26.9 的两个实用改动:LIMIT 支持 AFTER/UNTIL 条件边界,物化视图可 APPEND INCREMENTAL 增量刷新

官方发布说明:ClickHouse release 26.9

ClickHouse 26.9(2026-09-23)包含 56 个新特性、135 项性能优化、464 个 bug 修复。这里记录几处后端和数据工程里会直接用到的变化。

DateTime 与 Time 的算术运算

从 26.9 开始,可以对 DateTime 加上或减去 Time 值作为偏移量,结果保留 DateTime 的时区。

SELECT
    now() AS now,
    now + toTime('02:00:00'),
    toTime('04:00:00') + now,
    now64() AS now64,
    now64 - toTime('02:00:00'),
    now64 - toTime64('02:00:00.417', 3)
FORMAT Vertical;

输出:

Row 1:
──────
now: 2026-09-21 13:49:06
plus(now, to⋯02:00:00')): 2026-09-21 15:49:06
plus(toTime(⋯:00'), now): 2026-09-21 17:49:06
now64: 2026-09-21 13:49:06.440
minus(now64,⋯02:00:00')): 2026-09-21 11:49:06.440
minus(now64,⋯0.417', 3)): 2026-09-21 11:49:06.023

如果计算超出结果类型支持的范围,date_time_overflow_behavior 决定 ClickHouse 抛出异常、把结果钳制到最近边界,还是忽略溢出。DateTime 最大值是 2106-02-07 06:28:15。

SELECT toDateTime('2106-02-07 06:28:15', 'UTC') + toTime('00:00:01');
-- 默认 date_time_overflow_behavior = 'ignore'
-- 结果:1970-01-01 00:00:00(回绕到 DateTime 最小值)

SETTINGS date_time_overflow_behavior = 'throw' 时:

Code: 321. DB::Exception: Value 4294967296 is out of bounds of type DateTime ... (VALUE_IS_OUT_OF_RANGE_OF_DATA_TYPE)

SETTINGS date_time_overflow_behavior = 'saturate' 时返回该类型最大值 2106-02-07 06:28:15。

时区保留:

SELECT now() AS now, timezoneOf(now), now + toTime('02:00:00') AS future, timezoneOf(future) FORMAT Vertical;
-- now: 2026-09-21 14:37:21 / timezoneOf(now): Europe/London
-- future: 2026-09-21 16:37:21 / timezoneOf(future): Europe/London

PromQL 私有预览

26.9 扩展了 PromQL 支持:更多函数、更多 Prometheus HTTP API 端点、可直接对 TimeSeries 表做 SELECT 查询。PromQL 与 TimeSeries 表引擎现在在 ClickHouse Cloud 私有预览,可以把指标存进 ClickHouse,并从 ClickStack、Grafana、clickhouse-client 或 SQL 查询。详见 “Introducing ClickHouse's new TimeSeries engine” 博客。

CREATE TOKEN

26.9 引入 CREATE TOKEN:可以为应用、脚本、CI 任务、agent 创建限时凭证,而不用暴露或更换主密码。token 可限定为用户已有权限的子集,降低凭证泄露的影响。

CREATE TABLE ourTable ( id UInt8 );
INSERT INTO ourTable VALUES (1);

CREATE USER alexey IDENTIFIED WITH sha256_password BY 'main-password';
GRANT SELECT, INSERT ON ourTable TO alexey;
GRANT CREATE TOKEN ON *.* TO alexey;

以 alexey 连接,创建有效期 30 天、只能对 ourTable 执行 SELECT 的 token:

CREATE TOKEN
VALID FOR INTERVAL 30 DAY
GRANTS (SELECT ON ourTable);

返回:

┌─token────────────────────────────┬─────────valid_until─┐
│ NXuRnBywn4HcHCIyBWC2WHT2Cfn4xQLb │ 2026-10-21 15:07:57 │
└──────────────────────────────────┴─────────────────────┘

ClickHouse 只显示 token 一次,务必记下。如果不指定 VALID UNTIL 或 VALID FOR,默认有效期是 30 分钟。

用 token 连接执行 SELECT 正常:

./clickhouse client --user alexey --password 'NXuRnBywn4HcHCIyBWC2WHT2Cfn4xQLb' --query "SELECT * FROM ourTable"

用同一个 token 执行 INSERT 会被拒绝:

Code: 497. DB::Exception: alexey: Not enough privileges. To execute this query, it's necessary to have the grant INSERT(id) ON db.`table`. (ACCESS_DENIED)

token 永远不会赋予超过用户本身的权限;token 过期或用户被删除后即失效。

带条件边界的 LIMIT

26.9 扩展 LIMIT,支持按有序结果流中的值开始和停止输出。AFTER 包含匹配条件的行,UNTIL 在匹配行之前停止。可以加 ALL 让边界在每次匹配时都生效。

用于分析日志数据。先建表:

CREATE TABLE nginx_logs
(
    timestamp DateTime,
    ip String,
    method LowCardinality(String),
    path String,
    status UInt16,
    response_bytes UInt64,
    referer String,
    user_agent String
)
ENGINE = MergeTree
ORDER BY (timestamp, ip, path);

导入数据(s3 数据集):

INSERT INTO nginx_logs
WITH extractGroups(
    line,
    '^(\S+) - \S+ \[([^\]]+)\] "(\S+) (.*) [^ ]+" (\d+) (\d+) "([^"]*)" "(.*)"$'
) AS fields
SELECT
    assumeNotNull(parseDateTimeBestEffortOrNull(fields[2])) AS timestamp,
    fields[1] AS ip,
    fields[3] AS method,
    fields[4] AS path,
    toUInt16(fields[5]) AS status,
    toUInt64(fields[6]) AS response_bytes,
    fields[7] AS referer,
    fields[8] AS user_agent
FROM s3(
    'https://datasets-documentation.s3.eu-west-3.amazonaws.com/http_logs/nginx-66.log.gz',
    LineAsString
)
WHERE length(fields) = 8
  AND parseDateTimeBestEffortOrNull(fields[2]) IS NOT NULL;

数据概览:rows: 66514081、first_timestamp: 2019-01-24 00:00:00、last_timestamp: 2019-02-24 00:00:00、server_errors: 69857。

从第一个 5xx 开始返回五条请求(AFTER 是包含的):

SELECT timestamp, path, status
FROM nginx_logs
WHERE ip = '91.243.160.31'
  AND timestamp >= '2019-01-24 00:00:00'
  AND timestamp < '2019-02-03 00:00:00'
ORDER BY timestamp, ip, path, method, status, response_bytes
LIMIT 5 AFTER status >= 500;

结果从 2019-01-24 06:54:01 的 500 开始,后两条是 200。

UNTIL 是排他的,从第一个 5xx 开始、在第一个状态低于 500 的响应之前停止:

SELECT timestamp, path, status
FROM nginx_logs
WHERE ip = '91.243.160.31'
  AND timestamp >= '2019-01-24 00:00:00'
  AND timestamp < '2019-02-03 00:00:00'
ORDER BY timestamp, ip, path, method, status, response_bytes
LIMIT 100
AFTER status >= 500
UNTIL status < 500;

结果只有三条 500。

AFTER 边界默认只应用一次,加 ALL 会在每次有行匹配时重新应用。下面返回所有 5xx 以及其后的两条请求:

SELECT timestamp, path, status
FROM nginx_logs
WHERE ip = '91.243.160.31'
  AND timestamp >= '2019-01-24 00:00:00'
  AND timestamp < '2019-02-03 00:00:00'
ORDER BY timestamp, ip, path, method, status, response_bytes
LIMIT 3 AFTER status >= 500 ALL;

结果出现三段独立故障(第 2、7、12 行),每段内每个 500 都会重新应用三行边界,因此窗口会一直延伸到最后一个 5xx 之后连续两条非 5xx 请求。ALL 也可以重新应用起始边界,同时用 UNTIL 终止每段:

SELECT timestamp, path, status
FROM nginx_logs
WHERE ip = '91.243.160.31'
  AND timestamp >= '2019-01-24 00:00:00'
  AND timestamp < '2019-02-03 00:00:00'
ORDER BY timestamp, ip, path, method, status, response_bytes
LIMIT 100 AFTER status >= 500 ALL UNTIL status < 500;

这次输出排除了每段故障后的成功请求:UNTIL 在第一个非 5xx 响应处关闭区间,而 ALL 继续扫描下一个故障段。

增量式可刷新物化视图

26.9 给 refreshable materialized view 增加了 APPEND INCREMENTAL:不再每次刷新都全表扫描,只处理上次刷新以来提交的行。可用于把 append-only 数据增量复制到另一张 ClickHouse 表,或把 MergeTree 的事件流复制到 Iceberg 数据湖。

示例:在 ClickHouse 建 append-only 订单事件流,定期复制进 Iceberg 表。

CREATE TABLE order_events
(
    event_id UInt64,
    event_time DateTime,
    order_id UInt64,
    event_type LowCardinality(String),
    country LowCardinality(String),
    amount Decimal(10, 2)
)
ENGINE = MergeTree
ORDER BY (event_time, order_id, event_id)
SETTINGS
    enable_block_number_column = 1,
    enable_block_offset_column = 1,
    add_minmax_index_for_block_number_column = 1,
    add_minmax_index_for_block_offset_column = 1,
    part_minmax_index_columns = 'with_block_number_offset';

block-number 和 block-offset 列提供游标,ClickHouse 用它识别上次刷新之后提交的行。

启用 Iceberg 插入并创建本地 Iceberg 表:

SET allow_insert_into_iceberg = 1;
CREATE TABLE lake_order_events
(
    event_id UInt64,
    event_time DateTime,
    order_id UInt64,
    event_type String,
    country String,
    amount Decimal(10, 2)
)
ENGINE = IcebergLocal('lake_order_events', 'Parquet');

创建每小时把新提交事件复制到 Iceberg 的物化视图:

CREATE MATERIALIZED VIEW order_events_to_iceberg
REFRESH EVERY 1 HOUR APPEND INCREMENTAL
TO lake_order_events
AS
SELECT event_id, event_time, order_id, event_type, country, amount
FROM order_events;

插入数据后手动触发刷新:

SYSTEM REFRESH VIEW order_events_to_iceberg;
SYSTEM WAIT VIEW order_events_to_iceberg;

SELECT * FROM lake_order_events;

四条记录都已同步。再插入三条变更事件(shipped、refunded、placed),手动刷新后 Iceberg 表出现全部 7 行,证明每次刷新只处理新提交的行。

复制全文 生成海报 ClickHouse 数据库 SQL 物化视图 Iceberg

推荐文章

程序员茄子在线接单