Quick-Commerce-Real-Time-Analytics-Pipeline

KO | EN

gitlite — search

#react #ai #go #golang #android #rust #macos #awesome #cli #awesome-list #vue #pytorch #security #open-source

Quick-Commerce-Real-Time-Analytics-Pipeline

(★ 1)

Real-time streaming analytics platform for quick commerce — Kafka → Databricks (Bronze/Silver/Gold) → Synapse → Power BI, tracking live SLA breaches, rider availability & demand.

Quick-Commerce-Real-Time-Analytics-Pipeline 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Quick Commerce Real-Time Analytics Pipeline

A streaming data pipeline that ingests live order, delivery, product, and rider events for a quick-commerce (10-minute delivery) platform, processes them through a medallion architecture on Azure Databricks, and exposes business-ready metrics through Synapse serverless SQL.

## Problem Statement

Quick-commerce operations generate a constant stream of events — orders placed, riders going on/off duty, deliveries completing on time or late — and the business needs to see this in near real time, not in a next-day batch report. This project builds an end-to-end pipeline that ingests that event stream, cleans and structures it, and turns it into queryable metrics like on-time delivery rate, product popularity, and rider availability.

## Architecture

## Tech Stack

| Layer | Technology | Purpose |
|---|---|---|
| Event generation | Python + Faker | Simulates order/delivery/product/rider events |
| Streaming ingestion | Apache Kafka (Confluent Cloud) | Durable event bus between the app and the lakehouse |
| Processing | Azure Databricks (Structured Streaming) | Reads Kafka topics, transforms data through Bronze → Silver → Gold |
| Storage | Azure Data Lake Storage Gen2 + Delta Lake | Versioned, ACID-compliant storage for every layer |
| Governance | Unity Catalog | Table/schema governance across the lakehouse |
| Secrets | Azure Key Vault | Kafka credentials, never hardcoded in notebooks |
| Serving | Azure Synapse Analytics (Serverless SQL) | SQL views over the Gold/Silver Delta tables |
| BI (planned) | Power BI | Dashboarding on top of Synapse views |

## Data Flow

1. **Event generation** — Python scripts use Faker to simulate four event types: orders, deliveries, products, and riders.
2. **Kafka ingestion** — Events are published to four Confluent Kafka topics (one per event type) via `kafka_producer.py`.
3. **Bronze layer** — Databricks Structured Streaming jobs read each Kafka topic and write the raw payload (key, value, partition, offset, timestamps) into Delta tables with zero transformation — this preserves the original event exactly as received.
4. **Silver layer** — Each Bronze stream is parsed against a defined schema (`utils/schemas.ipynb`), null/malformed records are filtered out, and an `ingestion_time` column is added.
5. **Gold layer** — Silver tables are aggregated into business metrics:
   - **Business Metrics**: on-time vs. delayed deliveries in 3-hour windows, per city (with watermarking to handle late-arriving events)
   - **Order Status**: current status count across all orders
   - **Product Popularity**: units sold per product
   - **Rider Status**: current availability breakdown
6. **Serving layer** — Synapse serverless SQL creates views directly on top of the Gold/Silver Delta tables using `OPENROWSET`, so the data is queryable via standard SQL without a separate copy step.

## Key Engineering Decisions

- **Medallion architecture (Bronze/Silver/Gold)** — keeps raw data immutable and auditable while still producing clean, aggregate-ready tables for consumption.
- **Structured Streaming with checkpointing** — every stream (Bronze writes, Silver transforms, Gold aggregations) uses a dedicated checkpoint location, so jobs can resume from exactly where they left off after a failure or cluster restart.
- **Watermarking on Gold aggregations** — the business metrics view uses a 6-hour watermark on a 3-hour window to tolerate late-arriving delivery events without the state store growing unbounded.
- **Secrets via Key Vault, not hardcoded** — Kafka credentials are pulled at runtime through a Databricks service credential pointing at Azure Key Vault; nothing sensitive lives in the notebooks themselves.
- **Serverless SQL over a dedicated warehouse** — since this is an analytics-on-demand use case rather than constant heavy querying, Synapse serverless keeps cost proportional to actual usage.

## Dashboard

Not yet built — see "What I understand but didn't implement here" below for how I'd approach connecting Power BI to the Synapse views.

---

## Good to Know

**What I actually built** (verifiable in this repo):
- Synthetic event generators for 4 event types with realistic field distributions (`1_data_generation/events/`)
- A Kafka producer with delivery-confirmation callbacks (`1_data_generation/kafka_producer.py`)
- Three full Structured Streaming layers (Bronze/Silver/Gold) across 4 data domains, each with its own checkpointing (`2_databricks/`)
- Schema-enforced parsing at the Silver layer, including windowed + watermarked aggregation logic in Gold
- Synapse serverless SQL views exposing all Gold/Silver tables (`3_synapse_analytics/`)
- Secrets management via Azure Key Vault + Databricks service credentials (no hardcoded credentials anywhere in the pipeline)

**What I understand but didn't implement here** (adjacent production concepts I can discuss but shouldn't claim as shipped):
- *Power BI dashboard* — here's how I'd approach it: connect Power BI directly to the Synapse serverless SQL endpoint using DirectQuery mode, so the dashboard reflects the Gold tables without a separate data refresh step.
- *CI/CD for Databricks jobs* — here's how I'd approach it: use Databricks Asset Bundles to version-control job definitions and deploy notebooks across dev/staging/prod workspaces via a GitHub Actions pipeline.
- *Orchestration/scheduling* — here's how I'd approach it: wire the Bronze → Silver → Gold notebooks into a Databricks Workflow with task dependencies, so Silver only kicks off once Bronze's checkpoint has advanced.
- *Data quality monitoring* — here's how I'd approach it: add Great Expectations or Delta Live Tables expectations at each layer boundary to catch schema drift or null spikes before they reach Gold.
- *Alerting on pipeline failures* — here's how I'd approach it: hook Databricks job failure webhooks into a Slack/Teams channel so a broken stream gets flagged immediately instead of silently stopping.

---

## Setup

1. Create the required Azure resources (Storage Account, Key Vault, Databricks workspace, Synapse workspace) — see project setup notes.
2. Populate a `.env` file in `1_data_generation/` (see `.env.example`) with your Kafka credentials.
3. Run the event generators to start publishing to Kafka.
4. Deploy the notebooks in `2_databricks/` to your Databricks workspace and start the Bronze → Silver → Gold streams in order.
5. Run `3_synapse_analytics/sql_queries.sql` against your Synapse serverless SQL pool to create the views.

Quick-Commerce-Real-Time-Analytics-Pipeline

Quick-Commerce-Real-Time-Analytics-Pipeline 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Quick-Commerce-Real-Time-Analytics-Pipeline

Quick-Commerce-Real-Time-Analytics-Pipeline 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README