industrial-lakehouse-pipeline

KO | EN

gitlite — search

#javascript #python #hacktoberfest #ai #typescript #go #android #machine-learning #deep-learning #macos #java #kubernetes #mcp #open-source

industrial-lakehouse-pipeline

(★ 1)

A production-grade, local file-based data lakehouse implementing a "Medallion Architecture" (Bronze ➔ Silver ➔ Gold) to process multi-asset industrial IoT sensor feeds.

industrial-lakehouse-pipeline Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Industrial IoT Time-Series Pipeline & Lakehouse Ingestion

A production-grade, local file-based data lakehouse implementing a "Medallion Architecture" (Bronze ➔ Silver ➔ Gold) to process multi-asset industrial IoT sensor feeds.

The pipeline uses **Polars** for multi-threaded, high-performance time-series windowing and **DuckDB** as the embedded relational analytical engine to manage table lineages natively over Parquet files.

---

## 🏗️ Architecture & Core Components

- **Medallion Processing Blueprint:**
  - 🥉 **Bronze Tier:** Captures the raw, out-of-order incoming sensor Parquet streams including system anomalies, null frames, and duplicates.
  - 🥈 **Silver Tier:** Performs data cleaning using Polars. Enforces schema structures, fills sensor dropout values with historical median metrics, applies vectorised clip constraints to bounds, and computes a 10-minute time-window rolling mean.
  - 🥇 **Gold Tier:** Aggregates processed data points into highly structured hourly operational KPIs (averages, maximums, and rolling anomaly tallies) optimized for business intelligence views and predictive machine learning models.
- **Embedded Catalog Strategy:** Uses an in-process **DuckDB** instance to link underlying storage layers dynamically via relational views and schemas without database overhead.

---

## 📂 Project Structure

```text
industrial-lakehouse-pipeline/
│
├── storage/               # File-based Data Lakehouse Footprint
│   ├── bronze/            # Raw landing Parquet files from sensor ingestion
│   ├── silver/            # Sanitized, windowed flat time-series records
│   └── gold/              # Aggregated plant hourly analytical data profiles
│   └── lakehouse.db       # Embedded DuckDB database catalog
│
├── src/
│   ├── __init__.py
│   ├── database.py        # Relational schemas and catalog connections
│   ├── generator.py       # Multi-asset sensor stream simulator 
│   └── pipeline.py        # Core Polars-engineered transformation logic
│
├── main.py                # Pipeline driver execution script
├── requirements.txt       # Project dependencies
└── README.md              # Project documentation

```

---

## 🚦 Getting Started

### 1. Installation

Clone the repository and install the high-performance data engineering libraries:

```bash
pip install -r requirements.txt

```

### 2. Run the Full Lakehouse Orchestrator

Execute the main entry script to dynamically simulate the sensor stream, mount the storage catalog, and run the full end-to-end ELT lifecycle:

```bash
python main.py

```

---

## 📋 Data Pipeline Transformation Logic

| Architecture Tier | Engine Used | Strategy & Transformations Implemented |
| --- | --- | --- |
| **Bronze** | Python / Polars | Generates chaotic telemetry records across 5 industrial assets with simulated dropouts. |
| **Silver** | Polars / DuckDB | • Deduplicates timestamps per asset via `unique()`. <br>

<br>• Implements `.fill_null()` using running median asset calculations. <br>

<br>• Uses `.rolling_mean_by()` partition windows over time to extract trend features. |
| **Gold** | Polars / DuckDB | Groups clean signals into an `hourly_window` to compute plant metrics (`avg_temperature`, `max_vibration`, `total_anomalies`). |

industrial-lakehouse-pipeline

industrial-lakehouse-pipeline Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

industrial-lakehouse-pipeline

industrial-lakehouse-pipeline Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README