transaction-stream-data-pipeline

KO | EN

gitlite — search

#hacktoberfest #react #ai #golang #machine-learning #rust #linux #docker #nodejs #macos #java #windows #awesome-list #open-source

transaction-stream-data-pipeline

(★ 29)

Transaction processing & vis pipeline using PySpark Streaming

transaction-stream-data-pipeline 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Project overview

![transactions_stream_project_diagram](https://raw.githubusercontent.com/divakaivan/transaction-stream-data-pipeline/main/project-png/transactions_stream_project_diagram.png)

* **Stripe**: Using Stripe's API as the source of generating realistic transaction data.

* **Apache Kafka**: Stripe transaction data is streamed into Apache Kafka. It handles the data streams and ensures they are processed in real-time.

* **Apache ZooKeeper**: ZooKeeper is used alongside Kafka to manage and coordinate the Kafka brokers. ZooKeeper helps maintain configuration information, naming, synchronization, and group services.

* **PySpark**: The data from Kafka is then processed using PySpark Structured Streaming. This involves transforming individual transaction data into rows fit for a database.

* **Forex API**: GBP/x exchange rates are taken from an online API and updated every 24 hours.

* **PostgreSQL**: After processing, the data is stored in PostgreSQL.

* **dbt (Data Build Tool)**: dbt is used to manage and transform data within PostgreSQL. Data is split into dimension and a fact tables.

* **Grafana**: Finally, the data stored in PostgreSQL is visualized using Grafana.

# Data model

![transactions_stream_data_model](https://raw.githubusercontent.com/divakaivan/transaction-stream-data-pipeline/main/project-png/transactions_stream_data_model.png)

# dbt documentation

[Link to the docs](https://transaction-stream-data-docs.netlify.app/)

#### dbt lineage

# Visualisation

# Considerations for improvements

* add PySpark tests
* use an orchestrator
* use more data
  * Spark might be an overkill due to the data amount limitations, but I wanted to learn how to set Spark Streaming up in case data is much more  
  * for a better Grafana visualisation
  * maybe find an alternative transactions data source because the Stripe API has a 25 rate limit
  * also many of the generated values in a transaction from the Stripe API are null

# Setup

1. `git clone https://github.com/divakaivan/transaction-stream-data-pipeline.git`
2. Rename `sample.env` to `.env` and fill in the necessary environment variables
3. Type `make` in the terminal to see the setup options
```bash
Usage: make [option]

Options:
  help                 Show this help message
  build                Build docker services
  start                Start docker services (detached mode)
  stop                 Stop docker services
  dbt-test             Run dbt with LIMIT 100
  dbt-full             Run dbt with full data
```

transaction-stream-data-pipeline

transaction-stream-data-pipeline 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

transaction-stream-data-pipeline

transaction-stream-data-pipeline 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README