docs: add SeaTunnel 3.0 dev integration guide - #464
zhupitertop wants to merge 27 commits into
Conversation
dosubot
Bot
added
size:L
cf93179 to
4ae78ff
Compare
dosubot
Bot
added
size:M
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: The PR adds an unrelated release workflow that uses obsolete artifact rules, and its SeaTunnel guide contains unsupported claims and non-runnable configuration. Evidence: actionlint /tmp/validate-release-new-464.yml; exact-head patch; and comparison with the parent release workflow.
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: The new release workflow still fails valid artifacts or validates them incompletely, and the SeaTunnel documentation has unsupported compatibility and reference claims. Evidence: exact-head workflow/source review, actionlint, macOS grep reproduction, and current Apache documentation checks at 4ae78ff.
dosubot
Bot
added
size:L
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: The Chinese introduction page contains an unmatched fenced code block, so the rest of the page can render as code and its navigation/content is broken. Evidence: exact-head source has five triple-backtick fences with the extra opener at line 41; the latest deploy check is green but does not catch this Markdown structure.
imbajin
changed the title
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Final review on a136574: +1. The final diff is scoped to the Chinese SeaTunnel guide and its diagram, aligns the Source/Sink examples with current Next/dev behavior, documents the cross-graph ID constraints, and passes the Hugo build. Independent review score: 9.3/10, with no Critical or Important findings.
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: The quick-start mixes the released 2.3.13 distribution with dev-only HugeGraph connector APIs and contains several non-runnable Docker, Kafka, and migration examples. Evidence: the 2.3.13 HugeGraphSinkFactory requires schema_config; official SeaTunnel Docker/Source/Kafka documentation and exact-head source checks confirm the remaining incompatibilities.
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: The graph2graph prerequisite wording conflicts with the exact dev connector contract and should be corrected before users follow the migration steps. Evidence: the linked f1a1a0a SeaTunnel HugeGraph Sink documentation and the exact-head guide at line 259.
bitflicker64
left a comment
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: The guide matches upstream SeaTunnel on option names, defaults, client versions and dev status, but the section 4.2 edge example cannot pass startup validation against the schema declared in section 3.1, the 3.1 PropertyKey table is missing since, the CN introduction page loses its ecosystem overview, and a PR_DESCRIPTION.md scratch file is committed at the repository root. Evidence: upstream connector docs at tag 2.3.13 and at commit f1a1a0a, plus SchemaValidator, HugeGraphSink, HugeGraphSinkFactory and the mappers at 2.3.13. Not executed: no SeaTunnel, HugeGraph, MySQL or Kafka instance was started and Hugo was not run locally, so findings 3 to 5 come from reading the connector source rather than from an observed job failure.
docs: add Apache SeaTunnel connector integration guide (CN only) Add HugeGraph-SeaTunnel Connector documentation as an ecosystem entry point for users who want to sync data between HugeGraph and external systems via Apache SeaTunnel. Changes: - Add CN SeaTunnel connector doc in quickstart/toolchain/ - Add SeaTunnel to toolchain table and architecture diagram in CN introduction page - Follow existing Spark-connector documentation style and structure Co-Authored-By: Claude <noreply@anthropic.com> @
docs: add SeaTunnel integration architecture diagrams Add GPT-generated illustrations for the SeaTunnel Connector documentation: - Architecture diagram: HugeGraph + SeaTunnel data integration overview - Source/Sink diagram: bidirectional data flow between HugeGraph and external systems - Full-stack ecosystem diagram: complete HugeGraph toolchain visualization Placed in connector doc and introduction page for visual enhancement. Co-Authored-By: Claude <noreply@anthropic.com> @
docs: fix PR review issues for SeaTunnel connector doc Address reviewer feedback: - B1: Fix SeaTunnel installation link (/docs/start-v2/ -> /docs/getting-started/) - B2: Replace fake config params (url/graph/label) with real connector parameters (host/port/graph_name/mappings) from SeaTunnel docs - C: Add version compatibility matrix, note Source connector is dev-only - D: Add English docs synchronization (connector doc + introduction page) - E: Fix ASCII diagram border alignment (all lines now 70 chars wide) Co-Authored-By: Claude <noreply@anthropic.com> @
fix: correct version matrix and connector doc link - Java: 8+ -> 11+ (HugeGraph Client 1.5.0+ requires Java 11) - SeaTunnel: 2.3.12+ -> 2.3.13+ (bundles HugeGraph Client 1.5.0) - HugeGraph Server: 1.0.0+ -> 1.5.0+ (match client version in connector) - Link: generic /docs/connector-v2/ -> specific HugeGraph Sink page Co-Authored-By: Claude <noreply@anthropic.com> @
- add a single-node import walkthrough with verification - document HugeGraph Source reads and graph clone flows - align examples and limits with SeaTunnel dev - remove unrelated workflow and obsolete assets
- add Loader vs SeaTunnel selection guide for new users - add Kafka / Flink+Spark engines / graph migration scenarios with collapsible sections - add mermaid and ASCII diagrams per scenario - align config options with connector-hugegraph source - point SeaTunnel doc links to latest pages (Source doc links to dev branch file)
- move version matrix to top and deduplicate - add Docker-first deployment guide (K8s link, binary as reference) - replace FakeSource examples with LocalFile CSV - collapse engine section to doc links - add graph_space to all examples - add multi-graph batch migration example - drop license section, unnumbered references at end
- use schema_config (2.3.13 API) in quick-start and Kafka examples, scope mappings/Source to dev preview - fix Docker path: note JDK8 image and missing plugins, add JDK11 Dockerfile sample, correct doc link - fix container networking guidance (host.docker.internal / docker network) - fix Kafka option topics -> topic - fix host:port duplication in migration examples - correct check_vertex description and diagram output props - mark batch migration as pseudocode with per-table binding note
- clarify Tools, Loader, and SeaTunnel selection\n- align JDBC, Kafka, and dev migration examples\n- add concise diagrams and fold long configs\n- replace stale architecture asset and version links
8956f3b to
735e623
Compare
bitflicker64
left a comment
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: At this head the option keys in the JDBC, Kafka and HugeGraph Sink/Source examples match upstream, but the graph2graph edge migration now combines a PRIMARY_KEY vertex migration with raw ~source_id/~target_id passthrough. Edges are then only correct when person happens to get the same schema ID in both graphs, and the cross-graph ID caveat from the earlier revision was dropped. Separately, the section 7 table's heading does not match its rows. Evidence: static review of 735e6239 against apache/seatunnel f1a1a0a (EdgeMapper.buildVertexId/coerceRawVertexId, HugeGraphSourceReader lines 540-542, HugeGraphOptions.CHECK_VERTEX default false) and 2.3.13 (HugeGraphOptions, HugeGraphSinkFactory, KafkaBaseOptions, KafkaSourceOptions, JdbcCommonOptions), HugeGraph SplicingIdGenerator.generate, and the page at PR revision a13657421f. Not executed: no SeaTunnel, HugeGraph, MySQL or Kafka instance was started and Hugo was not run; no check runs are reported for this head.
- Recognize the Chinese-only SeaTunnel guide in language validation - Preserve the existing fail-closed fallback allowlist - Exercise language fallback scoping for the new guide
- complete the release schema and explain endpoint validation - preserve vertex IDs in dev graph migration and check endpoints - separate version-specific options and remove inert properties - remove the stray PR description file
- use current dev mappings for all five tasks - add setup steps, sample rows and result checks - document Zeta flushing and strict write failures - retain 2.3.13 as a sink-only compatibility note
- illustrate ingestion and migration on 3.0 dev - distinguish 2.3.13 sink-only support in the figure - replace four redundant assets with one overview
- retain all existing toolchain names in the overview - identify SeaTunnel source and sink as 3.0 dev features - keep the connector guide linked from the Chinese entry
imbajin
changed the title
- map both SQL endpoint columns to the name primary key - correct the fail-fast default from the pinned source - validate the document edge mapping against the dev validator
- allow the Chinese SeaTunnel page in the full validator - verify the actual standalone validation path - retain fallback rejection for unlisted untranslated pages
- compare graph import jobs with reusable data pipelines - state shared JDBC, Kafka and mapping capabilities - retain Loader Spark and Flink CDC context - summarize when Tools fits better
- contrast Loader imports with reusable SeaTunnel pipelines - show actual records becoming vertices and edges - illustrate preserved IDs across graph migration - close with a Tools, Loader and SeaTunnel decision guide
- use the selected dual-route comparison overview - align record mapping and ID migration illustrations - restyle the final tool selection diagram - retain all example data and version boundaries
- make all four diagrams open their original image - explain the full-size view for small field labels - preserve all task configurations and examples
- keep one explicit dev branch version note - simplify repeated version labels and reference links - remove redundant dev wording from troubleshooting text
bitflicker64
left a comment
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: The option names, defaults, reserved columns, build step and links in the guide match SeaTunnel at 35b2716, but the all-labels note names the mapping binding key source_table, and the Sink parser only accepts sourceTable. Evidence: static review of the exact head against connector-hugegraph MappingConfig/HugeGraphSinkOptions and seatunnel-api ConfigUtil at 35b2716; external links checked with curl; site validator tests run at head and base.
- correct the multi-label source option spelling - add a runnable sourceTable example - keep the source limitation note explicit
- add the English quick start for the pinned dev commit - mirror the Chinese scenarios, limits, and sourceTable note - add the English ecosystem entry and shared diagrams
- drop the Chinese-only SeaTunnel hreflang exception - restore strict versioning and site validation tests - rely on the synchronized English page for language parity
bitflicker64
left a comment
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: The option names, defaults, build steps and links match SeaTunnel at 35b2716, but the read-all-labels note gives sourceTable = "person", and the read-all Source never produces that table path, so a Sink using it fails at startup. Evidence: static review of the exact head against HugeGraphSourceFactory.buildReadAllTables, CatalogTableUtil.getCatalogTable, TablePath.getFullName, MappingConfig.appliesTo and HugeGraphSinkWriter at 35b2716; external links checked with curl. Not executed: no SeaTunnel or HugeGraph instance was started.
|
|
||
| > **为什么保留 ID？** HugeGraph 的 `PRIMARY_KEY` ID 包含顶点标签的内部 ID，两张图可能不同。例如源图顶点是 `1:marko`，目标图重新按主键生成的可能是 `2:marko`。如果重新生成顶点 ID 后仍复用源图的边端点，边就会连错。本例将原 ID 保存为字符串，因此会改变目标图的 ID 策略。 | ||
|
|
||
| 若要一次读取全部标签，省略 Source 的 `label` 后会按 `label_type` 为每个标签输出一张表；这时需用 `sourceTable` 将各 Sink 映射绑定到对应表，例如 `sourceTable = "person"`。不能直接套用本节的单标签配置。其他限制见 [HugeGraph Source 文档](https://github.com/apache/seatunnel/blob/35b2716cde7d4c91a24fc618a8d9cae90e213db3/docs/zh/connectors/source/HugeGraph.md)。 |
There was a problem hiding this comment.
sourceTable = "person" does not match what the read-all Source produces at 35b2716. HugeGraphSourceFactory.buildReadAllTables builds each label's table with CatalogTableUtil.getCatalogTable(label, rowType), which delegates to getCatalogTable("schema", "default", null, label, rowType). getTablePath().toString() then joins the non-null parts, so the table path is default.person. HugeGraphSinkWriter binds a mapping with MappingConfig.appliesTo, an exact String.equals, and when no mapping with a sourceTable matches it throws No mapping matched writer table 'default.person'. A reader following this example fails at job startup. Please change the example to sourceTable = "default.person", or say that the value must be the full table path shown in the writer log or error.
|
|
||
| > **Why preserve IDs?** A HugeGraph `PRIMARY_KEY` ID contains the internal ID of the vertex label, and that internal ID can differ between graphs. For example, a source vertex can be `1:marko`, while regenerating the primary key in the target graph can produce `2:marko`. Reusing the source edge endpoints after regenerating vertex IDs can connect edges to the wrong vertices. This example stores the original ID as a string, which changes the target graph's ID strategy. | ||
|
|
||
| When Source reads every label, omit `label` and it outputs one table per `label_type`. In that case, bind each Sink mapping to its table with `sourceTable`, for example `sourceTable = "person"`. Do not reuse the single-label configuration from this section. See the [HugeGraph Source documentation](https://github.com/apache/seatunnel/blob/35b2716cde7d4c91a24fc618a8d9cae90e213db3/docs/en/connectors/source/HugeGraph.md) for other limitations. |
There was a problem hiding this comment.
default.<label> (CatalogTableUtil.getCatalogTable(label, rowType) uses database default, and TablePath.getFullName() joins the non-null parts). MappingConfig.appliesTo compares with String.equals, so sourceTable = "person" matches no writer and HugeGraphSinkWriter fails with No mapping matched writer table 'default.person'. Please use sourceTable = "default.person", or say that the value must be the full table path. Also, "it outputs one table per label_type" should be one table per label; the Chinese text and upstream docs/en/connectors/source/HugeGraph.md both say one output table per label.
4 participants
变更
新增面向 SeaTunnel 3.0+ dev 分支 的中英文 HugeGraph 数据集成指南。所有示例固定到提交
35b2716，对应3.0.0-SNAPSHOT；并明确 2.3.13 只有 Sink，不能直接运行本文示例。name；多标签 Source 使用合法的sourceTable；迁移保留顶点 ID，先顶点后边；Kafka 使用 Zeta 的定时刷新配置。技术修复、较大改写、英文同步和 CI 清理分别提交，便于溯源与回滚。介绍页保留完整生态范围；仓库中的
PR_DESCRIPTION.md已删除。指定 review comment 已修复并已回复。图文结构
四张英文图统一采用浅网格、紫蓝线条的技术图风格，支持点击查看原图。按“总—分—总”组织，各自回答一个问题：工具如何区分 → 记录如何变成图 → 迁移如何保留端点 → 最后如何选型。
查看页面修改前后截图
验证
ac453a2e2：Hugo 0.165.0 Extended 的 production / staging latest 构建及版本链接校验通过；源文档链接、中英文标点和git diff --check通过。ac453a2e2已通过，包含六个版本构建及生产/预览站点汇总校验；PR 不执行发布步骤。验证边界：Java 测试使用实际 dev 相关源码、HugeGraph 1.7.0、SeaTunnel 2.3.13 兼容基础 API，并模拟服务端元数据。未运行完整 dev 引擎与真实 HugeGraph/MySQL/Kafka 端到端集成，也未验证完整 SeaTunnel 发行包构建。