pi-macos-harness

KO | EN

gitlite — search

#php #css #git #cli #rag #c #ai #cloud-native #nodejs #pytorch

pi-macos-harness

(★ 9)

A Pi extension for controlling macOS applications through macOS Harness

File Explorer

ZIP Download

- settings.json
- pre-commit
- - index.ts
- - LICENSE
  - SKILL.md
.editorconfig
.gitignore
LICENSE
package-lock.json
package.json
README.md
vite.config.ts

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

A [Pi](https://github.com/earendil-works/pi-mono) extension for controlling already-running native, Electron, and browser applications on macOS through [macOS Harness](https://github.com/browser-use/macos-harness).

The extension registers one sequential `macos_harness` tool. It executes bounded Python programs with `mac`, `browser`, `Path`, and `subprocess` preloaded, truncates large output, supports cancellation and timeouts, and automatically attaches screenshots returned by `mac.see()`.

## Install from GitHub

Install the Python runtime, then install this Pi package directly from GitHub:

```bash
uv tool install --python 3.12 --upgrade macos-harness
pi install git:github.com/HazAT/pi-macos-harness
```

Restart Pi or run `/reload`.

This project is distributed as a Git-backed Pi package and is not published to npm.

## Permissions

Check permissions without prompting:

```bash
macos-harness doctor
```

macOS Harness may need Accessibility, Screen Recording, and event-posting permission. Input Monitoring is not required. Only request missing permissions after the user explicitly approves macOS prompts:

```bash
macos-harness doctor --request
```

When Pi runs inside a terminal host such as super.engineering, grant permissions to that host application and restart it so macOS applies the new TCC state. Apple Events Automation is requested separately for each target application.

Anonymous upstream telemetry is enabled by default in macOS Harness. Disable it with:

```bash
macos-harness telemetry disable
```

## Use

Ask Pi to interact with an already-running Mac application, for example:

```text
Take a screenshot of the Finder window without bringing it forward.
Open the existing Slack tab in Chrome and capture it.
Inspect Spotify, search for an artist, and verify the result.
```

The agent receives the bundled `macos-harness` skill, which teaches it to:

- run `mac.doctor()` before first use
- prefer exact AppleScript or browser CDP operations when available
- bundle deterministic operations into one Python process
- stop at genuine decision boundaries
- preserve app focus and physical cursor position
- ask before requesting permissions or performing consequential actions

A direct tool call looks like:

```json
{
  "code": "frame = mac.see('Finder')\nprint(frame)",
  "timeout_seconds": 30
}
```

Images produced by `mac.see()` are attached to the tool result automatically.

## Local development

Development uses the globally installed [Vite+](https://viteplus.dev) CLI. Clone the repository, install its development dependencies through `vp`, and install the checked-in commit hook once:

```bash
git clone git@github.com:HazAT/pi-macos-harness.git
cd pi-macos-harness
vp install
vp config --no-agent
```

Run code-quality checks through Vite+:

```bash
vp check        # formatting and linting
vp check --fix  # apply safe formatting and lint fixes
vp fmt --check  # formatting only
vp lint         # linting only
```

Load the local source explicitly when testing outside the checkout:

```bash
pi -e ./pi-extension/macos-harness/index.ts \
  --skill ./skills/macos-harness/SKILL.md
```

The checked-in `.pi/settings.json` mirrors this setup for trusted local sessions. It disables any globally installed copy of the package while loading the extension and skill from the checkout, preventing duplicate resources during development.

Validate that Pi can load the extension without starting an agent turn:

```bash
pi --no-extensions \
  -e ./pi-extension/macos-harness/index.ts \
  --no-skills \
  --list-models '__load_check__'
```

Validate the skill from a pi-config checkout that contains the skill validator:

```bash
uv run ~/.pi/agent/skills/skill-creator/scripts/quick_validate.py \
  ./skills/macos-harness
```

## Releases

Invoke the manual-only local release skill from a trusted checkout:

```text
/skill:release patch
/skill:release 0.2.0
```

The workflow requires a clean worktree, runs `vp check` and a Pi package load check, updates both package version files, creates a conventional release commit and SemVer tag, pushes them, and publishes GitHub release notes containing pinned and latest install commands.

## How it works

Each tool call starts one `macos-harness` Python process. Within that process:

- Core Graphics captures windows and posts PID-targeted input events.
- Apple Accessibility exposes semantic elements and actions.
- Apple Events provide exact, focus-safe application commands.
- Browser Harness connects to the real logged-in Chrome session over CDP.
- A click-through AppKit overlay renders a virtual pointer without moving the physical cursor.
- Ordinary Python, files, and subprocesses remain available for one-off coordination logic.

The extension instruments `mac.see()` to report generated image paths back to Pi, then returns those PNGs as native image content blocks. Tool execution is sequential so independent model tool calls cannot race while controlling the desktop.

## Attribution

The runtime and original skill design come from [Browser Use's macOS Harness](https://github.com/browser-use/macos-harness), licensed under MIT. The adapted skill retains the upstream license in `skills/macos-harness/LICENSE`.

## License

MIT

pi-macos-harness

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

pi-macos-harness

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README