simple_dbt_project - GitLite

KO | EN

gitlite — search

#javascript #llm #golang #android #linux #macos #nodejs #kubernetes #awesome-list #vue #pytorch #security #ios #mcp

simple_dbt_project

(★ 185)

Code for dbt tutorial

파일 탐색기

최종 버전 다운로드 (.zip)

- devcontainer.json
- .gitkeep
- .gitkeep
- order_status_code.csv
- properties.yml
- .gitkeep
- - bronze.yml
  - bronze_clickstream.sql
  - bronze_customer.sql
  - bronze_orders.sql
  - bronze_state.sql
  - raw.yml
- - obt.yml
  - orders_obt.sql
- - fct_clickstream.sql
  - fct_orders.sql
  - silver.yml
- exposures.yml
- clickstream.csv
- clickstream_incremental.csv
- customer.csv
- customer_new.csv
- orders.csv
- state.csv
- .gitkeep
- dim_customer.sql
- .gitkeep
- customer_dimension_row_count_reconciliation_test.sql
.gitignore
.python-version
.tool-versions
dbt_project.yml
extract_load_pipeline.py
load_new_data.py
packages.yml
profiles.yml
pyproject.toml
qa_script.sh
README.md
uv.lock

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/josephmachado/simple_dbt_project

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd simple_dbt_project

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install .

PyPI에 배포된 패키지를 바로 설치합니다. 소스 클론이 필요 없습니다.

python <실행할 파일명>.py # README에서 정확한 실행 파일명을 확인하세요

파이썬 스크립트(또는 모듈)를 실행합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

* [dbt(data build tool) Tutorial](#dbtdata-build-tool-tutorial)
    * [CodeSpaces Setup (Recommended)](#codespaces-setup-recommended)
        * [Prerequisites](#prerequisites)
    * [Setup](#setup)
        * [Prerequisites](#prerequisites-1)
    * [Run pipeline with dbt](#run-pipeline-with-dbt)
    * [Validating SCD2 and Incremental table](#validating-scd2-and-incremental-table)

# dbt(data build tool) Tutorial

Code for blog **[dbt(data build tool) Tutorial](https://www.startdataengineering.com/post/dbt-data-build-tool-tutorial/)**

## CodeSpaces Setup (Recommended)

### Prerequisites

1. [GitHub Account](https://github.com/)

[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/josephmachado/simple_dbt_project)

> [!NOTE]
> Wait about 5 minutes for all packages to be automatically installed

## Setup

### Prerequisites

1. [uv](https://docs.astral.sh/uv/)
2. [git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)

Clone and cd into the repo as shown below:

```bash
git clone https://github.com/josephmachado/simple_dbt_project.git
cd simple_dbt_project
curl https://install.duckdb.org | sh # install duckdb cli 
curl -LsSf https://astral.sh/uv/install.sh | sh # install uv
```

## Run pipeline with dbt

Let's simulate the EL process with this Python script.

```bash
rm *.duckdb
uv run python extract_load_pipeline.py
```

> [!NOTE]
> We use `uv run` to run all our python commands. As this command with run the python process inside the `uv` virtual env.

We clean up any old dbt package dependencies and re-install them.

```bash
uv run dbt clean
uv run dbt deps
```

We use `seed` to load a [mapping file](./data/order_status_code.csv).

```bash
uv run dbt seed
```

We run the bronze layer pipeline first. Following that we run the `snapshot` pipeline which creates our `dim_customer` table. Finally we run the silver and gold layers.

```bash
uv run dbt run --select models/bronze
uv run dbt snapshot
uv run dbt run --select models/silver models/gold
```
We run pipelines in this order, since SCD2 tables can only be created with the `snapshot` command,

Finally we run the tests, create docs and serve them at port 8080.

```bash
uv run dbt test
uv run dbt docs generate
uv run dbt docs serve
```

Go to [http://localhost:8080](http://localhost:8080) to see the dbt documentation.

If you are running this on GitHub CodeSpaces, click on the `ports` tab and click on the link exposing port 8080.

## Validating SCD2 and Incremental table

Let's first check the current state of the SCD2 and Incremental tables.

```bash
uv run duckdb dbt.duckdb
```

```sql
select * from snapshots.dim_customer where customer_id = 82; -- one row 
select count(*) from fct_clickstream; -- 100
```

Now let's run the SCD2 and Incremental pipelines.

```bash
uv run python load_new_data.py # Inserts new data 
uv run dbt run --select models/bronze 
uv run dbt snapshot 
uv run dbt run --select models/silver
uv run dbt test
```

```bash
uv run duckdb dbt.duckdb
```

```sql
select * from snapshots.dim_customer where customer_id = 82; -- two row 
select count(*) from fct_clickstream; -- 110
```

> [!CAUTION]
> Do not forget to stop your codespaces machine