olist-lakehouse

KO | EN

gitlite — search

#javascript #react #typescript #llm #golang #go #android #nodejs #java #windows #awesome #kubernetes #awesome-list #vue

olist-lakehouse

(★ 9)

Production-grade Databricks PySpark lakehouse on the Brazilian E-Commerce (Olist) dataset, medallion architecture, 3 Spark MLlib models, MLflow tracking

olist-lakehouse 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Databricks PySpark Capstone - End-to-End E-Commerce Analytics Pipeline

> A production-grade medallion-architecture analytics platform on the Brazilian E-Commerce (Olist) dataset: PySpark ingestion, Delta Lake storage, Spark MLlib modeling, MLflow tracking, and a polished Streamlit dashboard - all backed by 140 green tests.

![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12-blue)
![Tests](https://img.shields.io/badge/tests-140%20passing-brightgreen)
![Lint](https://img.shields.io/badge/lint-ruff-blue)
![License](https://img.shields.io/badge/license-MIT-green)

---

## Elevator Pitch

This capstone transforms 1.5M+ raw e-commerce records into governed Delta tables, trained ML models, and an interactive dashboard. It demonstrates the full modern data stack end-to-end: ingestion, cleaning, aggregation, feature engineering, model training, experiment tracking, visualization, CI, and browser-level testing.

Every layer is reproducible from a clean clone in under five minutes.

---

## Key Features

- **Medallion architecture** - Bronze (raw) -> Silver (clean) -> Gold (aggregates + features) on Delta Lake
- **Pure PySpark pipeline** - typed `StructType` schemas, idempotent writes, partition-aware reads
- **Three production ML models** - regression, classification, and unsupervised clustering with Spark MLlib
- **MLflow tracking** - parameters, metrics, and model artifacts for every run
- **Premium Streamlit dashboard** - 6 pages, Plotly charts, hover animations, navy/blue theme
- **140 automated tests** - unit, integration, and Playwright browser E2E - all green
- **CI-enforced quality** - GitHub Actions runs lint + unit + integration on every push
- **Reproducible from scratch** - `make install && make download-data && make pipeline && make dashboard`
- **Complete deliverables** - 27-page PDF report, 17-slide PPTX deck, 3 Databricks notebooks

---

## Architecture

Data flows left to right. Each layer is a Delta table, versioned and queryable. ML models read from Gold features and register runs in MLflow. The Streamlit dashboard reads Gold tables via `deltalake` for sub-second rendering.

---

## Dataset

The [Brazilian E-Commerce Public Dataset by Olist](https://www.kaggle.com/datasets/olistbr/brazilian-ecommerce) covers orders placed between 2016 and 2018 across multiple marketplaces.

| Table | Rows | Description |
|-------|-----:|-------------|
| `olist_orders` | 99,441 | Order headers with timestamps and status |
| `olist_order_items` | 112,650 | Line items, prices, freight |
| `olist_order_payments` | 103,886 | Payment method, installments, value |
| `olist_order_reviews` | 99,224 | 1-5 star reviews with comments |
| `olist_customers` | 99,441 | Customer IDs and ZIP-code regions |
| `olist_sellers` | 3,095 | Seller IDs and locations |
| `olist_products` | 32,951 | Product catalog with category and dimensions |
| `olist_geolocation` | 1,000,163 | ZIP-code to lat/long mapping |

Plus a product-category translation table. Total: ~1.55M rows across 9 files.

---

## Quickstart

### 1. Prerequisites

- Python 3.11 or 3.12 (>=3.11,<3.13)
- Java 17 (required by PySpark 3.5)
- [Kaggle API credentials](https://www.kaggle.com/docs/api) in `~/.kaggle/kaggle.json` for dataset download

### 2. Install

```bash
git clone <your-fork-url> DatabricksProject
cd DatabricksProject
make install
```

This creates a virtual environment and installs the project in editable mode with all pipeline, ML, dashboard, and testing dependencies.

### 3. Download the data

```bash
make download-data
```

Pulls the Olist dataset from Kaggle into `data/raw/` (approximately 125 MB).

### 4. Run the pipeline, then the dashboard

```bash
make pipeline     # Bronze -> Silver -> Gold -> train 3 ML models
make dashboard    # Launch Streamlit on http://localhost:8501
```

That's it. The dashboard opens on the Overview page with live metrics computed from the Gold layer.

---

## Dashboard Screenshots

The Streamlit app has six pages, each rendered from Gold Delta tables with Plotly charts on a clean white background and navy accent theme.

| | |
|---|---|
| ![Overview](https://raw.githubusercontent.com/DivyamTalwar/olist-lakehouse/main/screenshots/overview.png) | ![Orders](https://raw.githubusercontent.com/DivyamTalwar/olist-lakehouse/main/screenshots/orders.png) |
| **Overview** - KPIs, revenue trend, top categories | **Orders** - status funnel, delivery timing, geo heat |
| ![Customers](https://raw.githubusercontent.com/DivyamTalwar/olist-lakehouse/main/screenshots/customers.png) | ![Sellers](https://raw.githubusercontent.com/DivyamTalwar/olist-lakehouse/main/screenshots/sellers.png) |
| **Customers** - RFM segments, repeat-rate cohorts | **Sellers** - leaderboard, revenue per seller, ratings |
| ![Products](https://raw.githubusercontent.com/DivyamTalwar/olist-lakehouse/main/screenshots/products.png) | ![ML Results](https://raw.githubusercontent.com/DivyamTalwar/olist-lakehouse/main/screenshots/ml_results.png) |
| **Products** - category mix, price bands, returns | **ML Results** - model metrics, feature importance |

All six screenshots are also embedded in the project report PDF.

---

## Testing

140 automated tests across three layers. All green on every CI run.

| Layer | Count | Scope |
|-------|------:|-------|
| Unit | 96 | Schemas, Bronze ingest, Silver cleaning, Gold aggregations, ML pipelines, MLflow helpers |
| Integration | 5 | Full Bronze->Silver->Gold pipeline, idempotency, lineage |
| Playwright E2E | 39 | Every dashboard page, every click, every tab - real browser |
| **Total** | **140** | |

Run them:

```bash
make test              # unit + integration
pytest tests/e2e -v    # Playwright browser tests (requires dashboard running)
ruff check .           # lint (0 errors)
```

CI runs lint + unit + integration on every push via `.github/workflows/ci.yml`.

---

## Project Structure

```text
DatabricksProject/
├── .github/workflows/ci.yml       # CI pipeline (lint + unit + integration)
├── data/                          # Gitignored: raw CSVs, bronze/silver/gold Delta tables
├── docs/
│   ├── project_report.pdf         # 27-page professional report
│   ├── capstone_presentation.pptx # 17-slide deck
│   ├── generate_report.py         # PDF generator
│   └── generate_slides.py         # PPTX generator
├── notebooks/                     # Databricks exploration notebooks
├── screenshots/                   # Dashboard screenshots (embedded in PDF)
├── scripts/
│   ├── download_data.py           # Kaggle dataset downloader
│   ├── populate_test_data.py      # Populate local Delta from fixtures
│   ├── run_pipeline.py            # Run full Bronze -> Silver -> Gold
│   └── capture_screenshots.py     # Playwright screenshot capture
├── src/ecommerce_pipeline/
│   ├── config.py                  # Paths, constants
│   ├── schemas.py                 # 8 PySpark StructType schemas
│   ├── spark_session.py           # SparkSession factory
│   ├── bronze/ingest.py           # CSV -> Delta ingestion
│   ├── silver/                    # 8 cleaning modules
│   ├── gold/                      # Aggregations + ML features
│   ├── ml/                        # 3 model training pipelines
│   ├── tracking/mlflow_utils.py   # MLflow helpers
│   └── dashboard/                 # Streamlit app + 6 views + components
├── tests/
│   ├── unit/                      # 96 unit tests
│   ├── integration/               # 5 pipeline integration tests
│   ├── e2e/                       # 39 Playwright E2E tests
│   └── fixtures/                  # Small CSVs for unit tests
├── pyproject.toml                 # Package config + Ruff + Pytest config
├── Makefile                       # make install | test | dashboard | pipeline
└── PLAN.md                        # Original 16-week execution plan
```

---

## Machine Learning

Three models trained on Gold features with Spark MLlib, tracked in MLflow.

| # | Task | Algorithm | Metrics |
|---|------|-----------|---------|
| 1 | Delivery time prediction | Gradient Boosted Trees Regressor | RMSE, MAE, R2 |
| 2 | Review score classification | Random Forest Classifier | Accuracy, F1 |
| 3 | Customer segmentation (RFM) | KMeans | Silhouette score |

Each training run logs parameters, metrics, and the fitted model to the MLflow tracking store. The ML Results page of the dashboard reads the latest runs and renders metric cards plus feature-importance charts.

---

## Tech Stack

| Category | Tools |
|----------|-------|
| Platform | Databricks, Delta Lake |
| Language | Python 3.11 / 3.12 |
| Big Data | Apache Spark 3.5, PySpark |
| Storage | Delta Lake 3.2, `deltalake` reader |
| ML | Spark MLlib (GBT, Random Forest, KMeans) |
| Tracking | MLflow |
| Dashboard | Streamlit 1.56, streamlit-extras, streamlit-option-menu |
| Charts | Plotly |
| Testing | Pytest, Playwright |
| Lint | Ruff |
| CI | GitHub Actions |
| Packaging | Hatchling, `pyproject.toml` |

---

## Deploy to Databricks

One notebook runs the complete pipeline on Databricks Free Edition in about 5 minutes on serverless compute. No cluster config, no bundle setup - the notebook `pip install`s the `ecommerce-pipeline` package directly from GitHub.

**Quick path:**

1. Upload the 8 Olist CSVs to `/dbfs/FileStore/olist/raw/` (Databricks UI -> Catalog -> DBFS -> Upload).
2. Import [`notebooks/00_databricks_pipeline.py`](notebooks/00_databricks_pipeline.py) into your workspace (copy-paste or Git folder).
3. Attach to Serverless -> **Run all**.

The notebook handles install, Bronze/Silver/Gold materialization, and MLflow-tracked training of all 3 models. Full step-by-step with Unity Catalog volume alternative, screenshots, and demo script: [`docs/DATABRICKS_DEPLOYMENT.md`](docs/DATABRICKS_DEPLOYMENT.md).

---

## Documentation

- [Databricks Deployment Guide](docs/DATABRICKS_DEPLOYMENT.md) - end-to-end deployment on Databricks Free Edition
- [Project Report (PDF, 27 pages)](docs/project_report.pdf) - full write-up with 6 embedded dashboard screenshots
- [Capstone Presentation (PPTX, 17 slides)](docs/capstone_presentation.pptx) - executive deck
- [Execution Plan](PLAN.md) - original 16-week week-by-week plan
- [Notebooks](notebooks/) - Databricks exploration notebooks (data exploration, EDA, ML experiments, end-to-end pipeline)

---

## Roadmap

- **Real-time streaming** - Structured Streaming ingestion from Kafka for live order events
- **NLP on reviews** - topic modeling and sentiment analysis on the Portuguese review corpus
- **Customer Lifetime Value** - probabilistic BG/NBD + Gamma-Gamma models on top of the RFM features
- **Automated retraining** - scheduled MLflow runs with drift detection and champion/challenger promotion
- **Production Databricks Workflow** - multi-task job with Unity Catalog governance and alerting

---

## License

MIT. See [LICENSE](LICENSE) for details.
# olist-lakehouse

olist-lakehouse

olist-lakehouse 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

olist-lakehouse

olist-lakehouse 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README