frusta - GitLite

KO | EN

gitlite — search

#security #golang #deep-learning #docker #llm #rust #kubernetes #kotlin #tailwindcss #video

frusta

(★ 13)

S3-inspired object store. single-node, parallel chunk pipeline

File Explorer

ZIP Download

- - api.js
  - checks.js
  - options.js
  - payload.js
  - run.sh
  - upload-flow.js
- readme.md
- - migration_lock.toml
- schema.prisma
- - env.ts
- - prisma.ts
- - errorHandler.middleware.ts
  - notFound.middleware.ts
  - requestLogger.middleware.ts
- - apiError.ts
  - apiResponse.ts
  - asyncHandler.ts
- app.ts
- server.ts
- - upload.integration.test.ts
- - upload.controller.unit.test.ts
  - upload.schema.unit.test.ts
  - upload.service.unit.test.ts
- tests-description.md
.gitignore
.gitkeep
LICENSE
package-lock.json
package.json
prisma.config.ts
README.md
tsconfig.json
vitest.config.ts

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

Command Glossary

Commands referenced in this DOCs, explained below.

🔍

`npm install`

View Details ▼

Install Node packages.

npm {{[i|install]}}

Install dependencies listed in `package.json`:

npm {{[i|install]}} {{package_name}}@{{version}}

Download a specific version of a package and add it to the list of dependencies in `package.json`:

npm {{[i|install]}} {{package_name}} {{[-D|--save-dev]}}

Download the latest version of a package and add it to the list of dev dependencies in `package.json`:

🔍

`npm run`

View Details ▼

Run a script.

npm run

List available scripts:

npm run {{script_name}}

Run a script:

npm run {{script_name}} -- {{argument}} {{--option}}

Pass arguments to a script:

🔍

`npm test`

View Details ▼

This command is an alias of `npm run test`.

tldr npm run

View documentation for the original command:

🔍

`npx`

View Details ▼

This command is an alias of `npm exec`.

tldr npm exec

View documentation for the original command:

🔍

`size`

View Details ▼

Display the sizes of sections inside binary files.

size {{path/to/file}}

Display the size of sections in a given object or executable file:

size {{[-o|--radix=8]}} {{path/to/file}}

Display the size of sections in a given object or executable file in [o]ctal:

size {{[-d|--radix=10]}} {{path/to/file}}

Display the size of sections in a given object or executable file in [d]ecimal:

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

```text
______              _        
|  ___|            | |       
| |_ _ __ _   _ ___| |_ __ _ 
|  _| '__| | | / __| __/ _` |
| | | |  | |_| \__ \ || (_| |
\_| |_|   \__,_|___/\__\__,_|
```

`manual single-node chunking engine | s3-style upload semantics | resumable + idempotent`

Frusta is a manual, S3-style chunk upload engine built on Node.js + Express + Prisma.
It accepts out-of-order and parallel chunk uploads, supports resume, enforces idempotency, and merges to a final artifact using atomic file operations.

## What It Solves

- Upload large files reliably over unstable networks.
- Resume interrupted uploads without re-sending completed chunks.
- Prevent duplicate chunk corruption under retries/races.
- Keep merge and cleanup deterministic with explicit upload session states.

## Core Features

- Chunked upload lifecycle: `INITIATED -> UPLOADING -> COMPLETING -> COMPLETED | FAILED`
- Out-of-order chunk acceptance
- Parallel chunk upload support
- Resume support via uploaded-chunk introspection endpoint
- Duplicate chunk protection:
  - Filesystem guard (`chunkN` existence check)
  - DB uniqueness (`@@unique([uploadSessionId, chunkIndex])`)
  - `createMany(..., skipDuplicates: true)` idempotency
- Safe write path:
  - Chunk write to `.part` then atomic rename to `chunkN`
  - Final merge to `<uploadId>-<fileName>.part` then atomic rename to final file
- Automatic temp cleanup after successful merge
- Failure rollback in chunk ingest path (if DB write fails after file write, chunk file is removed)
- Zod request validation for body/query/params
- Centralized error and not-found middleware
- Request logging middleware with latency printouts
- Test-time in-memory Prisma fallback for fast deterministic integration testing

## Architecture

### System Diagram

```mermaid
%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#e1f5fe', 'primaryTextColor': '#01579b', 'primaryBorderColor': '#0288d1', 'lineColor': '#0288d1', 'secondaryColor': '#e8f5e9', 'tertiaryColor': '#fff3e0'}}}%%
graph TB
    subgraph "Client"
        C1[Browser / Mobile App / k6]
    end

subgraph "HTTP Layer"
        A1[Express App<br/>src/app.ts]
        A2[Upload Router<br/>src/modules/uploads/uploads.routes.ts]
    end

subgraph "Cross-Cutting Middleware"
        M1[requestLogger]
        M2[express.json]
        M3[notFound]
        M4[errorHandler]
    end

subgraph "Controller Layer<br/>src/modules/uploads/uploads.controller.ts"
        CTRL1[initiateController]
        CTRL2[chunkController]
        CTRL3[statusController]
        CTRL4[completedController]
    end

subgraph "Validation Layer<br/>src/modules/uploads/uploads.schema.ts"
        V1[Zod: incomingSandesha]
        V2[Zod: chunkQuery]
        V3[Zod: completedSandesha]
        V4[Zod: statusParams]
    end

subgraph "Service Layer<br/>src/modules/uploads/uploads.service.ts"
        S1[prepareUploadDir]
        S2[storeChunk<br/>stream pipeline + atomic rename]
        S3[deleteChunk<br/>rollback cleanup]
        S4[mergeChunks<br/>ordered stream merge + atomic rename]
    end

subgraph "Data Layer<br/>src/db/prisma.ts"
        D1[Prisma Client]
        D2[(PostgreSQL)]
        D3[(In-Memory Adapter<br/>Test Fallback)]
    end

subgraph "File System Storage"
        F1[uploads/temp/&lt;uploadId&gt;/chunkN]
        F2[uploads/final/&lt;uploadId&gt;-&lt;fileName&gt;]
    end

C1 -->|POST /uploads/initiate| A1
    C1 -->|POST /uploads/chunk| A1
    C1 -->|GET /uploads/:id/status| A1
    C1 -->|POST /uploads/complete| A1

A1 --> M2
    M2 --> M1
    M1 --> A2
    A2 --> CTRL1
    A2 --> CTRL2
    A2 --> CTRL3
    A2 --> CTRL4

CTRL1 --> V1
    CTRL2 --> V2
    CTRL4 --> V3
    CTRL3 --> V4

CTRL1 -->|create uploadSession| D1
    CTRL1 --> S1
    CTRL2 -->|findUnique uploadSession| D1
    CTRL2 -->|update status UPLOADING| D1
    CTRL2 --> S2
    CTRL2 -->|createMany skipDuplicates| D1
    CTRL2 -->|count uploadedChunks| D1
    CTRL3 -->|findUnique + findMany| D1
    CTRL4 -->|findUnique uploadSession| D1
    CTRL4 -->|count chunks vs totalChunks| D1
    CTRL4 -->|update COMPLETING| D1
    CTRL4 --> S4
    CTRL4 -->|update COMPLETED/FAILED| D1

S1 --> F1
    S2 -->|write .part → rename chunkN| F1
    S3 -->|rm chunkN / .part| F1
    S4 -->|read chunk0..N| F1
    S4 -->|write .part → rename final| F2
    S4 -->|rm temp dir| F1

D1 --> D2
    D1 -.-> D3

A2 --> M3
    M3 --> M4
    M4 --> C1
```

### Upload Session State Machine

```mermaid
stateDiagram-v2
    [*] --> INITIATED : POST /initiate
    INITIATED --> UPLOADING : first chunk arrives
    UPLOADING --> UPLOADING : chunk N stored
    UPLOADING --> COMPLETING : POST /complete<br/>all chunks verified
    COMPLETING --> COMPLETED : merge + atomic rename success
    COMPLETING --> FAILED : merge error / exception
    UPLOADING --> FAILED : unhandled error
```

### Data Flow Highlights

1. **Initiate**: Client → Router → `initiateController` → Zod validation → Prisma (`uploadSession.create`) → `prepareUploadDir` → Temp directory created.
2. **Chunk Ingest**: Client → Router → `chunkController` → Zod validation → Prisma (`uploadSession.findUnique` + `update`) → `storeChunk` (stream pipeline, atomic `.part` → `chunkN` rename) → Prisma (`uploadChunk.createMany` with `skipDuplicates`) → On DB failure, `deleteChunk` rolls back the file.
3. **Status / Resume**: Client → Router → `statusController` → Prisma (`uploadSession.findUnique` + `uploadChunk.findMany`) → Returns `uploadedChunks[]` array.
4. **Complete / Merge**: Client → Router → `completedController` → Zod validation → Prisma (`count` verification) → Status `COMPLETING` → `mergeChunks` (ordered stream pipeline into `.part`, atomic rename to final, recursive temp cleanup) → Prisma status `COMPLETED` (or `FAILED` on error).

## Upload Flow

1. `POST /uploads/initiate` creates an upload session row and a temp directory for `uploadId`.
2. `POST /uploads/chunk` stores raw chunk bytes (`application/octet-stream`) by `chunkIndex`.
3. Clients can upload chunks in any order, retry safely, and in parallel.
4. `GET /uploads/:uploadId/status` returns `uploadedChunks[]` so clients resume only missing indexes.
5. `POST /uploads/complete` verifies `uploadedChunks === totalChunks`, merges in index order, atomically renames final artifact, then deletes temp chunk directory.

## API

Base path: `/uploads`

### 1) Initiate Upload

- `POST /uploads/initiate`
- Body:

```json
{
  "fileName": "video.mp4",
  "fileSize": 734003200,
  "totalChunks": 128
}
```

- Success: `201`

```json
{
  "success": true,
  "statusCode": 201,
  "message": "upload session initiated",
  "data": { "uploadId": "uuid" }
}
```

### 2) Upload Chunk

- `POST /uploads/chunk?uploadId=<uuid>&chunkIndex=<int>`
- Headers: `Content-Type: application/octet-stream`
- Body: raw chunk bytes
- Success: `200`

```json
{
  "success": true,
  "statusCode": 200,
  "message": "chunk stored",
  "data": { "uploadedChunks": 42, "totalChunks": 128 }
}
```

### 3) Upload Status (Resume API)

- `GET /uploads/:uploadId/status`
- Success: `200`

```json
{
  "success": true,
  "statusCode": 200,
  "message": "upload status fetched",
  "data": {
    "uploadId": "uuid",
    "status": "UPLOADING",
    "uploadedChunks": [0, 1, 2, 5, 6],
    "totalChunks": 128
  }
}
```

### 4) Complete Upload

- `POST /uploads/complete`
- Body:

```json
{
  "uploadId": "uuid"
}
```

- Success: `200`
- Idempotent behavior: if already completed, returns `200` with `"upload already completed"`.

## Storage and Merge Semantics

- Root: `UPLOAD_ROOT` (default: `uploads`)
- Temp chunks: `uploads/temp/<uploadId>/chunk<index>`
- Final artifact: `uploads/final/<uploadId>-<fileName>`
- Chunk and final files are first written as `.part`, then atomically renamed.
- On merge success: temp chunk directory is recursively removed.

## Data Model (Prisma)

### `uploadSession`

- `id` (UUID, PK)
- `fileName`
- `fileSize` (`BigInt`)
- `totalChunks`
- `status` (`INITIATED | UPLOADING | COMPLETING | COMPLETED | FAILED`)
- `mergeStartedAt`, `mergeCompletedAt`
- `createdAt`, `updatedAt`
- Index: `status`

### `uploadChunk`

- `id` (UUID, PK)
- `uploadSessionId` (FK -> `uploadSession`, `onDelete: Cascade`)
- `chunkIndex`
- `size`
- `createdAt`
- Unique constraint: `(uploadSessionId, chunkIndex)`
- Index: `uploadSessionId`

## Project Structure

```text
src/
  app.ts                          # express app wiring + middleware + routes
  server.ts                       # process entrypoint
  config/env.ts                   # zod env validation
  db/prisma.ts                    # prisma client + in-memory test fallback
  middleware/
    requestLogger.middleware.ts
    notFound.middleware.ts
    errorHandler.middleware.ts
  modules/uploads/
    uploads.routes.ts             # upload route map
    uploads.schema.ts             # zod contracts
    uploads.controller.ts         # request orchestration
    uploads.service.ts            # fs/stream chunk + merge engine
    uploads.constants.ts          # status constants + storage paths
    uploads.types.ts
  utils/
    apiError.ts
    apiResponse.ts
    asyncHandler.ts

prisma/
  schema.prisma
  migrations/

tests/
  integration/upload.integration.test.ts
  unit/upload.controller.unit.test.ts
  unit/upload.service.unit.test.ts
  unit/upload.schema.unit.test.ts

benchmarks/
  k6/
    upload-flow.js
    api.js
    checks.js
    options.js
    payload.js
    run.sh
```

## Local Setup

1. Install dependencies:

```bash
npm install
```

2. Configure environment:

```bash
# .env
NODE_ENV=development
PORT=3000
DATABASE_URL=postgresql://<user>:<pass>@<host>:5432/<db>
# optional:
# UPLOAD_ROOT=uploads
```

3. Apply DB schema:

```bash
npx prisma migrate deploy
```

4. Run:

```bash
npm run dev
```

## Environment Variables

- `NODE_ENV`: `development | test | production`
- `PORT`: HTTP port (default `3000`)
- `DATABASE_URL`: required unless using in-memory test mode
- `UPLOAD_ROOT` (optional): storage root (default `uploads`)
- `FRUSTA_TEST_USE_REAL_DB` (optional): when `true` in test mode, bypasses in-memory DB fallback

## NPM Scripts

- `npm run dev`: run server in watch mode via `tsx`
- `npm run test`: run full Vitest suite
- `npm run typecheck`: TypeScript type-check
- `npm run build`: build to `dist/`
- `npm run start`: run compiled server
- `npm run bench:k6:upload`: run k6 flow with env-driven params
- `npm run bench:k6:upload:local`: convenience local benchmark profile

## Testing

Run:

```bash
npm test
```

Current snapshot (run on March 12, 2026):

- `29/29` tests passing
- Unit: `23` (controller + service + schema)
- Integration: `6` (full HTTP upload flow)

Notes:

- In `NODE_ENV=test`, DB defaults to in-memory Prisma adapter unless `FRUSTA_TEST_USE_REAL_DB=true`.
- Integration tests cover upload initiation, chunk ingest, out-of-order behavior, status/resume API, complete/merge path, and invalid index rejection.

## Benchmark (k6) Snapshot

Script: `benchmarks/k6/upload-flow.js`

Run profile used:

- Date: March 12, 2026
- `BASE_URL=http://127.0.0.1:3000`
- `VUS=5`
- `DURATION=15s`
- `CHUNK_SIZE=8`
- `SLEEP_SECONDS=0`
- Server mode: `NODE_ENV=test` (in-memory DB path)

Observed results:

- Checks pass rate: `100%` (`126/126`)
- HTTP failure rate: `0.00%`
- Total HTTP requests: `54`
- Throughput: `1.86 req/s`
- `http_req_duration` avg: `1664.49 ms`
- `http_req_duration` p90: `2081.15 ms`
- `http_req_duration` p95: `2654.46 ms`
- Completed upload iterations: `6`

Threshold status for this run:

- `http_req_failed < 1%`: pass
- `checks > 99%`: pass
- `http_req_duration p95 < 2500ms`: narrowly missed (`2654.46ms`)

## Why This Design Holds Up

- Stream-based writes (`pipeline`) reduce memory pressure and give cleaner backpressure handling.
- Atomic renames protect against partial file visibility.
- Session/chunk split in DB keeps metadata clean and queryable.
- Idempotent chunk semantics make retries safe under network noise and client duplication.
- Controller/service separation keeps the upload engine modular and easy to evolve.

## License

This project is licensed under the ISC License.
See [LICENSE](/home/amaan/my_stuff/frusta/LICENSE).