Skip to content

Commit 7ad0d92

Browse files
authored
perf(clickhouse): stop storing native event attributes (#4866)
## Summary Stop storing native JSON event attributes while retaining the existing materialized `attributes_text` representation. Also remove the two unused full-text indexes from the source event table. ## Design This depends on [#4860](#4860). Its explicit-column writer must be deployed before this migration is applied so attributes continue to feed the materialized string. The JSON input becomes EPHEMERAL, eliminating its stored subcolumns and merge overhead for new parts. Existing parts retain their JSON files until normal merges or TTL removal. The migration is intentionally not reversible automatically. Restoring native JSON storage safely requires inspecting the current schema and coordinating the writer change, so it should be done with a new forward migration rather than a retry-sensitive Down migration.
1 parent 17f29b7 commit 7ad0d92

2 files changed

Lines changed: 64 additions & 0 deletions

File tree

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
-- +goose Up
2+
3+
-- Full-text search is served outside the source event table. Keeping these
4+
-- indexes here adds work to every event insert and merge without serving reads.
5+
ALTER TABLE trigger_dev.task_events_v2
6+
DROP INDEX IF EXISTS idx_attributes_text_search;
7+
8+
ALTER TABLE trigger_dev.task_events_v2
9+
DROP INDEX IF EXISTS idx_message_text_search;
10+
11+
-- attributes remains an insert input for attributes_text, but is no longer
12+
-- stored. Writers must include attributes in an explicit insert column list
13+
-- because implicit INSERT column lists exclude EPHEMERAL columns.
14+
ALTER TABLE trigger_dev.task_events_v2
15+
MODIFY COLUMN attributes JSON EPHEMERAL;
16+
17+
-- +goose Down
18+
19+
-- Restoring the stored JSON column safely requires inspecting the live schema
20+
-- and coordinating the writer rollback. Use a new forward migration instead.
21+
SELECT throwIf(1, 'This migration cannot be rolled back automatically');

internal-packages/clickhouse/src/taskEvents.test.ts

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,49 @@ describe("task events v2", () => {
6262
has_inserted_at: 1,
6363
},
6464
]);
65+
66+
const readColumnKinds = ch.reader.query({
67+
name: "read-task-event-attribute-column-kinds",
68+
query: `SELECT name, default_kind, default_expression
69+
FROM system.columns
70+
WHERE database = 'trigger_dev'
71+
AND table = 'task_events_v2'
72+
AND name IN ('attributes', 'attributes_text')
73+
ORDER BY name`,
74+
schema: z.object({
75+
name: z.string(),
76+
default_kind: z.string(),
77+
default_expression: z.string(),
78+
}),
79+
});
80+
const [columnError, columns] = await readColumnKinds({});
81+
expect(columnError).toBeNull();
82+
expect(columns).toEqual([
83+
{
84+
name: "attributes",
85+
default_kind: "EPHEMERAL",
86+
default_expression: "defaultValueOfTypeName('JSON')",
87+
},
88+
{
89+
name: "attributes_text",
90+
default_kind: "MATERIALIZED",
91+
default_expression: "toJSONString(attributes)",
92+
},
93+
]);
94+
95+
const readRemovedIndexes = ch.reader.query({
96+
name: "read-removed-task-event-text-indexes",
97+
query: `SELECT name
98+
FROM system.data_skipping_indices
99+
WHERE database = 'trigger_dev'
100+
AND table = 'task_events_v2'
101+
AND name IN ('idx_attributes_text_search', 'idx_message_text_search')
102+
ORDER BY name`,
103+
schema: z.object({ name: z.string() }),
104+
});
105+
const [indexError, indexes] = await readRemovedIndexes({});
106+
expect(indexError).toBeNull();
107+
expect(indexes).toEqual([]);
65108
}
66109
);
67110
});

0 commit comments

Comments
 (0)