aethergrid

KO | EN

gitlite — search

#macos #tailwindcss #proxy #generative-ai #awesome-list #android #go #react-native #monitoring #swift

(★ 9)

A decentralized compute orchestration platform powered by strict Domain-Driven Design (DDD). Decoupled domain core, event-driven state reconciliation, and zero-overhead cluster management.

File Explorer

ZIP Download

- - ci.yml
- - __init__.py
- - __init__.py
  - database.py
- - __init__.py
  - api.py
  - auth.py
  - dependencies.py
  - request_logging_middleware.py
- main.py
- - 0001-use-psycopg-over-sqlalchemy.md
  - 0002-defer-psycopg-pool-open-deprecation.md
  - 0003-defer-frontend-bundle-optimization.md
  - 0004-worker-registration-api.md
  - 0005-worker-job-lease-ownership.md
  - 0006-worker-execution-loop.md
  - 0007-reconciliation-loop.md
  - 0008-event-repository-persistence.md
  - 0009-identifier-value-object-reconstruction.md
  - 0010-postgres-repository-parity.md
  - 0011-job-reclaim-and-reconciliation-repair.md
  - 0012-real-job-execution.md
  - 0013-job-listing-endpoint.md
  - 0014-continuous-lease-renewal.md
  - 0015-api-key-authentication.md
  - 0016-job-lifecycle.md
  - 0017-third-party-testclient-warning.md
  - 0018-domain-owns-scheduling-policy.md
  - 0019-standalone-worker-agent-process.md
  - 0020-expose-job-command-to-authenticated-agents.md
  - 0021-per-api-key-rate-limiting.md
  - 0022-structured-request-and-background-loop-logging.md
  - 0023-in-process-error-tracking.md
  - 0024-frontend-api-key-handling.md
  - 0025-issuing-production-api-keys-via-render-shell.md
  - 0026-fail-loudly-when-storage-backend-is-unset.md
  - 0027-worker-not-found-on-lease-acquisition.md
  - 0028-expose-job-command-through-public-create-job-api.md
  - 0029-cancel-running-job-via-lease-renewal.md
  - 0030-idempotent-worker-registration-per-node.md
  - 0031-reclaim-a-job-abandoned-mid-cancellation.md
  - 0032-move-cluster-tick-execution-off-the-event-loop.md
  - 0033-persist-job-started-immediately-after-worker-start.md
  - 0034-fence-lease-release-by-lease-identity.md
  - 0035-delete-lease-before-reclaiming-offline-node-jobs.md
  - 0036-fence-outcome-reports-by-lease-identity-for-agents.md
  - 0037-verify-lease-fencing-against-a-real-repository.md
  - 0038-fence-lease-renewal-by-lease-identity.md
  - 0039-fully-reclaim-abandoned-job-on-worker-reregistration.md
- - 01-system-overview.md
  - 02-clean-architecture.md
  - 03-backend-components.md
  - 04-frontend-components.md
  - 05-request-flow.md
  - 06-database.md
  - 07-future-cluster.md
  - README.md
- - logo.png
- - apple-touch-icon.png
  - favicon-16x16.png
  - favicon-32x32.png
  - icons.svg
  - logo.png
- - App.tsx
  - index.css
  - main.tsx
- .gitignore
- eslint.config.js
- index.html
- package-lock.json
- package.json
- postcss.config.js
- README.md
- tailwind.config.js
- tsconfig.app.json
- tsconfig.json
- tsconfig.node.json
- vite.config.ts
- vitest.config.ts
- issue_api_key.py
- preflight_demo_check.sh
- run_agent.py
- - test_create_job_endpoint.py
- - test_acquire_lease_service.py
  - test_assign_worker_service.py
  - test_authenticate_api_key_service.py
  - test_cancel_job_service.py
  - test_cluster_capacity_service.py
  - test_cluster_health_service.py
  - test_cluster_tick_service.py
  - test_cluster_utilization_service.py
  - test_complete_job_service.py
  - test_create_api_key_service.py
  - test_create_job_service.py
  - test_create_node_service.py
  - test_create_worker_service.py
  - test_error_tracking_service.py
  - test_fail_job_service.py
  - test_get_job_history_service.py
  - test_get_job_service.py
  - test_get_node_service.py
  - test_heartbeat_node_service.py
  - test_job_execution_service.py
  - test_list_available_nodes_service.py
  - test_list_nodes_service.py
  - test_list_offline_nodes_service.py
  - test_list_queued_jobs_service.py
  - test_mark_dead_workers_service.py
  - test_rate_limiter_service.py
  - test_record_job_events_service.py
  - test_register_worker_service.py
  - test_release_lease_service.py
  - test_release_lease_service_against_postgres.py
  - test_remove_offline_node_service.py
  - test_renew_lease_service.py
  - test_report_job_outcome_service.py
  - test_request_logging_service.py
  - test_retry_job_service.py
  - test_revoke_api_key_service.py
  - test_schedule_highest_priority_job.py
  - test_schedule_job_records_event.py
  - test_scheduler_loop_service.py
  - test_worker_heartbeat_service.py
- - test_api_key.py
  - test_constraint_aware_scheduler.py
  - test_event.py
  - test_job.py
  - test_job_constraints.py
  - test_job_lifecycle.py
  - test_job_lifecycle_service.py
  - test_job_priority.py
  - test_job_retry_policy.py
  - test_lease.py
  - test_node.py
  - test_node_heartbeat.py
  - test_node_labels.py
  - test_node_liveness.py
  - test_resourse_requirements.py
  - test_scheduler.py
  - test_scheduler_allocation.py
  - test_scheduler_best_fit.py
  - test_scheduler_ignores_draining_nodes.py
  - test_scheduler_multiple_constraints.py
  - test_worker_liveness.py
- - __init__.py
- - conftest.py
  - test_cancel_job_api.py
  - test_cluster_tick_does_not_block_event_loop.py
  - test_create_job_api.py
  - test_create_job_request_schema.py
  - test_create_node_api.py
  - test_get_job_api.py
  - test_get_job_history_api.py
  - test_get_worker_api.py
  - test_list_events_api.py
  - test_list_jobs_api.py
  - test_list_nodes_api.py
  - test_register_worker_api.py
  - test_renew_lease_api.py
  - test_report_job_outcome_api.py
  - test_request_logging_middleware.py
  - test_start_job_api.py
- - __init__.py
  - test_issue_api_key_guards_against_test_database.py
  - test_run_agent_renew_lease.py
- __init__.py
.dockerignore
.gitignore
docker-compose.yml
Dockerfile
pyproject.toml
README.md
SECURITY.md

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

A distributed AI workload orchestrator built around the problems that make scheduling hard at scale: exclusive execution ownership under failure, reconciliation after partial failures, and enforced resource limits. Not a CRUD tutorial with a scheduler theme.

AetherGrid takes workloads, matches them against available compute nodes based on resource requirements and constraints, and manages the full lifecycle: queued, scheduled, running, completed, failed, retried, cancelled. Jobs run through workers registered against nodes, and job execution ownership is enforced through time-bound leases rather than a simple assignment flag. Every route requires API key authentication, including the endpoint that issues keys.

**Try it live**: the full console is deployed and reachable at [aethergrid-dashboard.onrender.com](https://aethergrid-dashboard.onrender.com) with real Postgres, real auth, and real API-key-gated endpoints.

---

## Why This Exists

Most scheduler side-projects are a single `main.py` script wrapped in a `while True` loop polling an in-memory dictionary. They work fine, right up until you need to swap the persistence engine, add a new constraint type, or figure out why a job silently disappeared, or why two workers picked up the same job at once.

AetherGrid was built around one rule: **the domain logic doesn't know or care where the data lives.** Jobs, nodes, workers, and the allocation algorithm are pure Python with zero infrastructure dependencies. The database is a detail, not the foundation. This project is a concrete demonstration that these architectural patterns aren't just conference-talk vocabulary; they're guardrails that keep a codebase understandable as it grows, and as its correctness requirements get harder.

---

## Engineering Decision Records

Every non-obvious decision in this codebase, why a domain rule lives where it does, why an obvious-looking shortcut was rejected, what broke and how it got fixed, is written down at the moment it was made, not reconstructed afterward for a portfolio. 38 ADRs live in [`/docs/adr`](docs/adr). A few worth reading directly if you want to see the reasoning, not just the conclusion:

- [**ADR 0007 - Reconciliation Loop**](docs/adr/0007-reconciliation-loop.md): how the system detects and repairs state left inconsistent by dead workers and expired leases, instead of assuming the happy path is the only path.
- [**ADR 0011 - Job Reclaim and Reconciliation Repair**](docs/adr/0011-job-reclaim-and-reconciliation-repair.md): closing a real race condition where a dying worker's lease renewal could land after reconciliation had already started reassigning its work.
- [**ADR 0012 - Real Job Execution**](docs/adr/0012-real-job-execution.md): building genuine subprocess execution with enforced timeouts, then deliberately keeping it unreachable from the public API until the system had authentication, and proving that absence with a test rather than a comment.
- [**ADR 0014 - Continuous Lease Renewal**](docs/adr/0014-continuous-lease-renewal.md): why a lease is renewed continuously for a job's entire runtime instead of once at acquisition.
- [**ADR 0015 - API Key Authentication**](docs/adr/0015-api-key-authentication.md): why opaque server-issued tokens were chosen over JWTs for a system that needs instant revocation, and why building authentication still didn't answer whether `Job.command` should be exposed, that question stayed open until [ADR 0020](docs/adr/0020-expose-job-command-to-authenticated-agents.md), which resolved it narrowly: a worker can read the one command already assigned to it, nothing broader.
- [**ADR 0018 - Domain Owns Scheduling Policy**](docs/adr/0018-domain-owns-scheduling-policy.md): why `list_available()` moved out of the repository entirely, since deciding which nodes are eligible for scheduling is a business rule, not a persistence concern, and letting infrastructure decide that would have made scheduling behavior dependent on which database backend was running.
- [**ADR 0019 - Standalone Worker Agent Process**](docs/adr/0019-standalone-worker-agent-process.md): replacing in-process job execution with a real out-of-process agent that confirms its own execution start over the network, and why pull-based polling was chosen over push delivery, since it reuses reconciliation this codebase already trusts instead of introducing new failure and delivery-guarantee logic.
- [**ADR 0029 - Cancel a Running Job by Reusing Lease Renewal**](docs/adr/0029-cancel-running-job-via-lease-renewal.md): extending an existing, already-trusted channel (lease renewal) to deliver cancellation to a running job's subprocess, instead of adding a second polling mechanism, and introducing an explicit CANCELLING status so a job caught mid-cancellation is a visible, real state rather than indistinguishable from one still running.
- [**ADR 0030 - Idempotent Worker Registration Per Node**](docs/adr/0030-idempotent-worker-registration-per-node.md): keying worker identity to node identity so an agent restarting reclaims its existing worker instead of leaving a permanently dead row behind, using the same recovery path reconciliation already trusts, closing a gap before fleet-scale restart churn could ever expose it.
- [**ADR 0031 - Reclaim a Job Abandoned Mid-Cancellation**](docs/adr/0031-reclaim-a-job-abandoned-mid-cancellation.md): finalizing a job as CANCELLED, with no retry consumed, when its worker dies before confirming a cancellation already in progress, instead of leaving it stranded in CANCELLING forever or silently retrying a stop nobody asked to undo.
- [**ADR 0032 - Move Cluster Tick Execution Off the Event Loop**](docs/adr/0032-move-cluster-tick-execution-off-the-event-loop.md): running the cluster tick on a worker thread via asyncio.to_thread once real, caller-controlled commands could run for an unbounded duration, so one long-running job can no longer stall every other request the server would otherwise serve.
- [**ADR 0033 - Persist a Job's RUNNING Transition Immediately After worker.start()**](docs/adr/0033-persist-job-started-immediately-after-worker-start.md): closing a gap independently rediscovered twice, WorkerRepository.save() only ever writes the workers table, so a job's own row silently stayed SCHEDULED for its entire real execution unless something separately persisted it, extracting a single shared function instead of patching the second occurrence in isolation.
- [**ADR 0034 - Fence Lease Release by Lease Identity, Not Just Worker Identity**](docs/adr/0034-fence-lease-release-by-lease-identity.md): closing a gap one level upstream of what ADR 0014 had already named and deferred, releasing a lease checked only that a worker held some lease, not that it was the same one the caller had been renewing, letting a stale caller delete a different, legitimately-held lease out from under whoever actually owned it, verified with a test that proved the unsafe delete happening before any fix landed.
- [**ADR 0035 - Delete a Job's Lease Before Reclaiming It in RecoverOfflineNodeService**](docs/adr/0035-delete-lease-before-reclaiming-offline-node-jobs.md): the third independent occurrence of the same shape in one session, a sibling service (RecoverExpiredLeaseService) already had the correct pattern, and this one hadn't inherited it, silently stranding every job recovered from an offline node since its stale lease was never deleted, verified with a test that proved the strand before any fix landed.
- [**ADR 0036 - Fence Standalone-Agent Outcome Reports by Lease Identity**](docs/adr/0036-fence-outcome-reports-by-lease-identity-for-agents.md): closing the one limitation ADR 0034 named rather than claimed to cover, the external HTTP contract a standalone agent uses to report a job's outcome carried no lease identity at all, so a stale report could never be distinguished from a current one. `lease_id` is now a required field, captured by the agent at poll time before execution starts and checked against whatever lease is actually current for the worker, not looked up internally where it could never disagree with itself.
- [**ADR 0037 - Verify Lease Fencing Against a Real, Reconstructing Repository**](docs/adr/0037-verify-lease-fencing-against-a-real-repository.md): ADR 0034 and ADR 0036's lease-fencing logic had only ever been proven against `InMemoryLeaseRepository`, which shares object references on read, the same class of gap that produced a false-negative repro attempt during ADR 0033's own development before that bug was proven against SQLite instead. Adds a test reproducing the full lease-theft scenario, acquire, reconcile-reclaim, legitimate reassignment, stale release attempt, against `PostgresLeaseRepository`, with each step run through its own independently constructed repository instance so a shared in-memory reference can't mask the behavior under test.
- [**ADR 0038 - Fence Lease Renewal by Lease Identity, Not Just Worker Identity**](docs/adr/0038-fence-lease-renewal-by-lease-identity.md): the third and final lease-mutating operation to close this gap, after release (ADR 0034) and outcome reporting (ADR 0036). `RenewLeaseService` resolved the current lease by worker_id alone and renewed whatever it found, with no check that the caller's belief about which lease it held still matched reality. Unlike release, a mismatch here is silent rather than destructive: a stale caller renews the wrong lease with no error, potentially masking a real reconciliation failure. Closes both the internal WorkerExecutionLoop path and the external `POST /workers/{worker_id}/lease/renew` contract in the same change, since renewal is the standalone agent's repeated heartbeat mechanism, not a one-time terminal call like release.
- [**ADR 0039 - Fully Reclaim an Abandoned Job on Worker Re-registration**](docs/adr/0039-fully-reclaim-abandoned-job-on-worker-reregistration.md): ADR 0030 accepted, at the time, that reclaiming a worker on agent restart would silently discard its abandoned job, calling that a mirror of reconciliation's own recovery behavior. That mirror stopped being accurate the moment reconciliation was fixed to always release a node's resources and delete a stale lease before recovering a worker, a pairing worker re-registration never inherited. Left alone, this wasn't only a resource leak: since lease acquisition fences by job identity, not worker identity, and a recovered worker goes idle immediately, it could be reassigned a new job before the old one's lease expired, and that old lease's eventual expiry would then corrupt the new job's tracking instead. Fixed by reusing reconciliation's exact reclaim sequence, lease deletion, resource release, job reclaim, before recovering the worker, verified the same way as the gap it closed: a test proving the corruption before any fix landed.

If you're evaluating whether someone can operate at a systems level rather than a feature level, this is the fastest way to check.

---

## Key Capabilities

- **API key authentication gating every route**: no endpoint in the system, including the one that issues keys, is reachable without a valid credential. The only way to mint the first key is a script run locally with direct database access, never over HTTP, closing the exact self-service-credential hole that pattern would otherwise leave open
- **Job lifecycle management**: explicit state transitions (Queued → Scheduled → Running → Completed/Failed/Cancelled), including cancellation of a job already `RUNNING` via an explicit `CANCELLING` state, with configurable retry policies and priority-aware scheduling, plus cancel and retry actions reachable from the dashboard
- **Per-job lifecycle history**: every job has a dedicated detail page (`/jobs/{id}`) showing its full real event timeline, `JobCreated` through completion, not just its current status
- **Constraint-aware best-fit allocator**: matches workloads to nodes based on resource requirements and labels, while skipping nodes that are draining or offline
- **Node draining**: a healthy node can be taken out of scheduling rotation for maintenance without killing it outright; the scheduler stops assigning it new work while anything already running on it continues to completion
- **Worker registration and heartbeats**: registering a node automatically registers a worker against it, so it's immediately capable of claiming and executing work, not just existing as unused capacity
- **Standalone worker agent with exclusive job ownership**: `scripts/run_agent.py` runs as a real, separate process, polling the API over HTTP for assigned work, executing it as a real local subprocess, and heartbeating on its own background thread for the agent's entire lifetime, independent of whatever job it's currently executing (see [ADR 0019](docs/adr/0019-standalone-worker-agent-process.md)). This replaces the dashboard's client-side heartbeat as the liveness mechanism for any worker running it; a worker with no agent process attached still falls back to node liveness alone. Every worker is tagged with an explicit `managed_by` field set at registration (`DASHBOARD` or `AGENT`); the in-process scheduler loop skips any worker marked `AGENT` entirely, so a standalone agent's jobs are executed exactly once, by the agent, never raced against the in-process path.
- **Lease-based execution ownership**: when a worker accepts a job, it holds a renewable, expiring lease on that job, continuously renewed for the job's entire execution, so retries, reconnects, network failures, and jobs that simply run long can't result in two workers executing the same job
- **Explicit execution-start confirmation**: `POST /workers/{worker_id}/jobs/{job_id}/start` lets whatever is actually executing a job, the in-process scheduler loop for dashboard-managed workers, a standalone agent for agent-managed ones (ADR 0019), confirm execution has genuinely begun. This is the one call that transitions a job from `Scheduled` to `Running`; assignment alone no longer does (see [ADR 0019](docs/adr/0019-standalone-worker-agent-process.md))
- **Real subprocess execution with enforced timeouts**: jobs with a command run as real subprocesses, with a two-stage shutdown (graceful `SIGTERM`, then `SIGKILL` after a grace period) if a job overruns its execution timeout
- **Node liveness tracking**: heartbeat-based health checks, automatic detection of offline nodes, and resource reclamation when work fails or nodes disappear
- **Reconciliation with bounded retries**: jobs abandoned by a dead worker or an offline node are reclaimed back to the queue within their retry budget, and fail outright once that budget is exhausted, so a single unhealthy node can't cause a job to be reassigned and abandoned indefinitely, with the reclaim ordered to close a real race where a dying worker's renewal could land after reconciliation had already started reassigning its lease
- **Domain event recording**: every lifecycle transition a job goes through, `JobCreated`, `JobScheduled`, `WorkerAssigned`, `LeaseAcquired`, `LeaseReleased`, `JobCompleted`/`JobFailed`, and `JobReclaimed`, is persisted as an immutable event at the exact point it happens
- **Live cluster-wide event feed**: `GET /events` and a real-time Activity Feed on the dashboard, polling every 3 seconds, so the story an individual job tells on its own detail page is also visible as it happens across the whole cluster
- **Worker visibility**: a dedicated Workers table showing every registered worker, its status, the node it belongs to, what it's running, and when it was last seen
- **Multi-page dashboard**: real client-side routing (`/`, `/nodes`, `/jobs`, `/jobs/{id}`) instead of a single page, with active-route highlighting in the sidebar

---

## Architecture

The system is split into four layers, with dependencies pointing inward:

**Domain**: `Job`, `Node`, `Worker`, `Lease`, `Event`, and `ApiKey` aggregates enforce their own invariants. The scheduling algorithm and job lifecycle state machine live here as plain Python, with no imports from FastAPI or psycopg. Delete the infrastructure layer entirely and the domain tests still pass.

**Application**: Services such as `ScheduleJobService`/`SchedulerService`, `AssignWorkerService`, `AcquireLeaseService`, `StartJobService`, `DrainNodeService`, `ClusterHealthService`, and `AuthenticateApiKeyService` coordinate domain objects and repositories without embedding business rules that belong one layer down. A `WorkerExecutionLoop` drives a worker through executing its assigned job as a real subprocess, continuously renewing its lease on a background thread for the job's entire runtime, recording the real outcome, and releasing the lease regardless of that outcome. A renewal that fails means the lease has already been reclaimed elsewhere, and the loop discards its result rather than risk persisting it against another worker's in-progress or completed work. A `ReconciliationLoop` catches the failure modes the happy path can't: crashed workers, expired leases, state left inconsistent by infrastructure failures.

**Infrastructure**: PostgreSQL implementations exist for every repository (`Node`, `Job`, `Worker`, `Lease`, `Event`, `ApiKey`), written with raw `psycopg` instead of an ORM, a deliberate choice to keep query behavior and transaction boundaries visible rather than abstracted away. `Node`, `Job`, and `Event` additionally have SQLite implementations for local development; `ApiKey` deliberately does not, since local development already runs against the same PostgreSQL backend production uses, and a SQLite path would reintroduce the environment drift that consolidation was built to remove. Every repository is validated against a shared **contract test suite** run against each backend it supports, so switching between implementations, or trusting that they behave identically, is a tested guarantee rather than an assumption.

**Presentation**: FastAPI endpoints for jobs, nodes, workers, events, cluster health, and API keys that validate input, call an application service, and return a response. Every route, on every router, requires a valid API key. No business logic lives in this layer. The frontend mirrors the same discipline: `api/*.ts` typed HTTP calls, `hooks/*.ts` data-fetching hooks, and page/component composition, no business logic embedded in components either.

Every non-obvious decision, why domain owns scheduling instead of application, why raw psycopg over an ORM, how job lifecycle transitions are enforced, why leases exist instead of a simple assignment field, why renewal is a strict update rather than an upsert, why opaque tokens were chosen over JWTs, why job commands were deliberately kept unreachable from the public API until authentication existed, then reopened narrowly, first for workers reading only their own assigned job's command ([ADR 0020](docs/adr/0020-expose-job-command-to-authenticated-agents.md)), later for the public `CreateJobRequest` API itself ([ADR 0028](docs/adr/0028-expose-job-command-through-public-create-job-api.md)), is documented as an ADR in [`/docs/adr`](docs/adr).

---

## A Deliberate Security Decision Worth Naming

The execution engine runs real, arbitrary commands as subprocesses with real timeout enforcement. `Job.command` and `Job.exit_code` are fully tested at the service layer. Until recently, neither was exposed through the public `CreateJobRequest` API, a boundary enforced by a test asserting the field's absence, not left as a comment.

Shipping the capability before exposing it was the deliberate call: build it correctly, prove it works, defer the exposure decision until it can be made deliberately rather than by default (ADR 0012).

Authentication closed the first gap. Every route now requires a valid API key, and the only way to mint one without already holding one is a script run locally with direct database access, never over HTTP (ADR 0015). That left a sharper question open: what should *any* authenticated caller be trusted to do, given this system runs a single key tier with no role distinction.

That question has been answered twice, narrowly each time. ADR 0020 let an assigned worker read the one command already set for its own job, nothing broader. ADR 0028 closed the rest: `command` is now settable through the public API, gated by the same key requirement every other route uses. No new credential tier, by design, matching ADR 0018's stance against building permission systems for risk that hasn't been measured. In production, that risk is scoped to whoever holds a Render Shell-issued key (ADR 0025), today, exactly one person. ADR 0028 names the exact trigger for revisiting that: the day a second key exists.

The pattern holds throughout: build it right, prove it works, name the risk before shipping the exposure, not just wait for the last blocker to clear.

---

## Test Coverage

341 tests across domain, application, infrastructure, and API layers, all passing:

- Full domain logic coverage: job lifecycle, retry policy, constraint matching, node and worker liveness, lease semantics, node draining and the scheduler's exclusion of draining nodes, and API key issuance, revocation, and usage tracking
- Contract tests proving every repository's in-memory, SQLite (where implemented), and PostgreSQL implementations behave identically, including foreign-key-enforced aggregates such as `Worker` and `Lease`, and specifically that lease renewal fails rather than resurrects a lease already reclaimed by reconciliation
- Application service tests for every use case, including lease acquisition, renewal, release, reconciliation repair (both the requeue-with-retries-remaining path and the fail-outright-once-exhausted path), real subprocess execution (including tests that genuinely kill a process that ignores `SIGTERM`, forcing `SIGKILL`, on both the timeout path and the cancellation path), and the full API key lifecycle from issuance through revocation
- Event recording tests proving every lifecycle event fires at the correct point, in the correct order, across the full job lifecycle, scheduling, assignment, lease acquisition and release, completion, failure, and reconciliation reclaim
- API-level tests against real FastAPI endpoints, including the cluster-wide event feed, per-job history, and every route's auth requirement, verified through a real end-to-end request, not mocked

```bash
pytest
```

---

## Running It

```bash
git clone https://github.com/wycliffRotich-dev/aethergrid.git
cd aethergrid
docker compose up --build
```

This starts the API and a Postgres instance. Issue yourself a key before calling anything, every route requires one:

```bash
python scripts/issue_api_key.py "local-dev"
```

Run the frontend separately:

```bash
cd frontend
npm install
npm run dev
```

CI runs the full test suite against a live Postgres service on every push. See `.github/workflows`.

---

## Scope

This isn't trying to compete with Kubernetes or Ray at scale. It's a demonstration of how to build a system that stays understandable as it grows: layered correctly, tested honestly, and documented well enough that someone else could pick it up and know exactly why every piece is where it is, including the pieces that are deliberately half-built and marked as such.

---

Let's connect&nbsp;&nbsp;&nbsp;<a href="https://wa.me/254745275288"><img src="https://img.icons8.com/ios-filled/32/25D366/whatsapp.png" alt="WhatsApp" /></a>&nbsp;&nbsp;<a href="https://linkedin.com/in/rotichkipleting"><img src="https://img.icons8.com/ios-filled/32/0A66C2/linkedin.png" alt="LinkedIn" /></a>&nbsp;&nbsp;<a href="mailto:celestinerotich969@gmail.com"><img src="https://img.icons8.com/ios-filled/32/EA4335/gmail.png" alt="Email" /></a>

aethergrid

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README