Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions TOC-tidb-cloud-lake.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,8 @@
- [Load from Bucket](/tidb-cloud-lake/guides/load-from-bucket.md)
- [Load from Local File](/tidb-cloud-lake/guides/load-from-local-file.md)
- [Load from Remote File](/tidb-cloud-lake/guides/load-from-remote-file.md)
- Load with Platforms
- [Load with dbt](/tidb-cloud-lake/guides/load-with-dbt.md)
- Load Semi-structured Data
- [Overview](/tidb-cloud-lake/guides/load-semi-structured-data.md)
- [Load Parquet](/tidb-cloud-lake/guides/load-parquet.md)
Expand Down
30 changes: 16 additions & 14 deletions tidb-cloud-lake/guides/load-with-dbt.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
---
title: dbt
title: Load Data with dbt
summary: dbt is a transformation workflow that helps you get more work done while producing higher quality results. You can use dbt to modularize and centralize your analytics code, while also providing your data team with guardrails typically found in software engineering workflows. Collaborate on data models, version them, and test and document your queries before safely deploying them to production, with monitoring and visibility.
---

# dbt
# Load Data with dbt

[dbt](https://www.getdbt.com/) is a transformation workflow that helps you get more work done while producing higher quality results. You can use dbt to modularize and centralize your analytics code, while also providing your data team with guardrails typically found in software engineering workflows. Collaborate on data models, version them, and test and document your queries before safely deploying them to production, with monitoring and visibility.

[dbt-databend-cloud](https://github.com/databendcloud/dbt-databend) is a plugin developed by {{{ .lake }}} with the primary goal of enabling smooth integration between dbt and {{{ .lake }}}. By utilizing this plugin, you can seamlessly perform data modeling, transformation, and cleansing tasks using dbt and conveniently load the output into {{{ .lake }}}. The table below illustrates the level of support that the dbt-databend-cloud plugin offers for commonly used features in dbt:
[tidbcloudlake-dbt](https://github.com/tidbcloud/lake-dbt) is a plugin developed by {{{ .lake }}} with the primary goal of enabling smooth integration between dbt and {{{ .lake }}}. By utilizing this plugin, you can seamlessly perform data modeling, transformation, and cleansing tasks using dbt and conveniently load the output into {{{ .lake }}}. The table below illustrates the level of support that the tidbcloudlake-dbt plugin offers for commonly used features in dbt:

| Feature | Supported ? |
|----------------------------- |----------- |
Expand All @@ -19,36 +19,38 @@ summary: dbt is a transformation workflow that helps you get more work done whil
| Sources | Yes |
| Custom Data Tests | Yes |
| Docs Generate | Yes |
| Snapshots | No |
| Snapshots | Yes |
| Connection Retry | Yes |

## Installing dbt-databend-cloud
## Install tidbcloudlake-dbt

Installing the dbt-databend-cloud plugin has been streamlined for your convenience, as it now includes dbt as a required dependency. To effortlessly set up both dbt and the dbt-databend-cloud plugin, run the following command:
Installing the tidbcloudlake-dbt plugin has been streamlined for your convenience, as it now includes dbt as a required dependency. To effortlessly set up both dbt and the tidbcloudlake-dbt plugin, run the following command:

```shell
pip3 install dbt-databend-cloud
pip3 install tidbcloudlake-dbt
```

However, if you prefer to install dbt separately, you can refer to the official dbt installation guide for detailed instructions.

## Tutorial: Run dbt Project jaffle_shop

If you're new to dbt, {{{ .lake }}} recommends completing the official dbt tutorial available at <https://github.com/dbt-labs/jaffle_shop>. Before you start, follow [Installing dbt-databend-cloud](#installing-dbt-databend-cloud) to install dbt and dbt-databend-cloud.
If you're new to dbt, {{{ .lake }}} recommends completing the official dbt tutorial available at <https://github.com/dbt-labs/jaffle_shop>. Before you start, follow [Install tidbcloudlake-dbt](#install-tidbcloudlake-dbt) to install dbt and tidbcloudlake-dbt.

This tutorial provides a sample dbt project called "jaffle_shop," offering hands-on experience with the dbt tool. By configuring the default global profile (~/.dbt/profiles.yml) with the necessary information to connect to your {{{ .lake }}} instance, the project will generate tables and views defined in the dbt models directly in your {{{ .lake }}} database. Here's an example of the file profiles.yml that connects to a {{{ .lake }}} instance:

```yml title="~/.dbt/profiles.yml"
jaffle_shop_databend:
jaffle_shop_lake:
target: dev
outputs:
dev:
type: databend
host: 127.0.0.1
port: 8000
type: tidbcloudlake
host: tnxxxx.gw.aws-us-east-2.default.tidbcloud.com
port: 443
schema: sjh_dbt
user: databend
user: <username>
pass: ********
warehouse: default
secure: true
```

If you're using {{{ .lake }}}, you can refer to this [Wiki page](https://github.com/databendcloud/dbt-databend/wiki/How-to-use-dbt-with-Databend-Cloud) for step-by-step instructions on how to run the jaffle_shop dbt project.
For more information about configuring and using the adapter, see the [lake-dbt repository](https://github.com/tidbcloud/lake-dbt).
Loading