agentmote - GitLite

KO | EN

gitlite — search

#php #terminal #mcp #data-science #llm #graphql #cross-platform #gemini #csharp #ios

agentmote

(★ 9)

An open BLE hardware platform for ESP32 devices that pair with and control coding agents like Codex and Claude Code.

File Explorer

ZIP Download

- - CMakeLists.txt
  - codex_control.c
- - CMakeLists.txt
  - codex_ble_espidf.c
- - CMakeLists.txt
- - CMakeLists.txt
  - two_button_input.c
- - CHANGELOG.md
  - CMakeLists.txt
  - idf_component.yml
  - Kconfig
  - Kconfig.uac
  - license.txt
  - README.md
  - usb_device_uac.c
- - microphone_input.h
- CMakeLists.txt
- idf_component.yml
- Kconfig.projbuild
.codex
.gitignore
CMakeLists.txt
dependencies.lock
LICENSE
NOTICE.md
partitions_sticks3.csv
README.md
sdkconfig.defaults
sdkconfig.sticks3.defaults
version.txt

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

AgentMote is ESP-IDF firmware that turns an ESP32-S3 device into a Codex
Micro-compatible BLE controller and native USB microphone. It supports the
original four-button breadboard and M5Stack StickS3 from one codebase. StickS3
adds a task-status screen, a two-button gesture system, ES8311 microphone input,
and safe local speaker cues.

This is an independent, unofficial compatibility project. See
[NOTICE.md](NOTICE.md) before distributing a device or firmware image.

## What it does

- Advertises over BLE as `Codex Micro` using the compatible vendor HID report.
- Uses Just Works bonding and reconnects after a host disconnects.
- Maps GPIO47, GPIO0, GPIO39, and GPIO40 to the core Codex voice workflow.
- Answers the host RPC methods used for version, status, lighting, and focused
  application synchronization.
- Exposes a portable semantic control API for Codex actions, joystick, battery,
  connection, and host events.
- Uses a bounded 4096-byte receive buffer and validates report fragment sizes.
- Captures an INMP441 over I2S and enumerates the ESP32-S3 native USB port as
  `AgentMote Microphone` (UAC 2.0, mono, 48 kHz, 16-bit).
- Keeps microphone audio and Codex controls independent: audio uses USB, while
  buttons and host RPC continue to use BLE.
- On StickS3, renders BLE/USB/battery state and all six host Agent task states.
- Adds semantic Fast, Fork, and Agent 1–6 controls plus a two-button command
  palette.

ESP32-S3 does not support the Bluetooth Classic headset profiles, and its BLE
stack does not expose a standard wireless microphone to macOS. AgentMote
therefore uses the native USB/OTG port for reliable, driverless audio.

## M5Stack StickS3

StickS3 uses its built-in display, ES8311 codec, MEMS microphone, speaker,
250 mAh battery, and buttons on GPIO11 and GPIO12. No external wiring is needed.
The power/reset button retains the M5PM1 factory behavior.

Home-screen controls:

| Input | Action |
| --- | --- |
| Key1 single click | Send |
| Key1 double click | Approve |
| Key1 hold, then release | Push-to-talk press/release |
| Key2 single click | Select next assigned Agent slot |
| Key2 double click | Open guarded Decline confirmation; Key2 confirms, Key1 cancels |
| Key2 hold | Fork current chat |
| Press both keys | Open command palette |

The command palette contains Fast, Fork, and Agent 1–6. Key1 moves to the next
entry and Key2 runs it. Hold Key1 or press both keys to close it. The palette
closes after 8 seconds of inactivity; Decline confirmation cancels after 2
seconds.

Local tones acknowledge connection and successful/cancelled actions. Speaker
volume is capped well below the hardware maximum and tones are suppressed while
push-to-talk is active. The firmware intentionally exposes no USB speaker.

When USB-C is connected, macOS can use `StickS3 Codex Microphone` as a mono
48 kHz input while Codex controls remain on BLE. With USB disconnected, BLE
controls continue to work, but audio comes from whichever microphone macOS or
ChatGPT has selected; firmware cannot change the host input device itself.

## Breadboard hardware

Target: ESP32-S3 with the pin assignment from Xiaozhi
`bread-compact-wifi`.

The handheld layout is ordered from top to bottom:

| Position | GPIO | Codex action | Gesture |
| --- | ---: | --- | --- |
| Top | 40 | Decline | Hold for at least 700 ms, then release |
| Upper middle | 39 | Approve | Press and release |
| Lower middle | 0 / BOOT | Send | Press and release |
| Bottom | 47 | Mic | Hold to talk, release to stop |

Each normally-open switch connects its GPIO to GND when pressed. The firmware
enables the ESP32-S3 internal pull-ups; do not connect a switch to 3.3 V or 5 V.
GPIO0 still selects the ROM download mode when held during reset. At runtime it
acts as Send, and startup input handling waits for it to be released before
accepting button events.

Connect the INMP441 using the same layout as Xiaozhi `bread-compact-wifi`:

| INMP441 | ESP32-S3 | Notes |
| --- | ---: | --- |
| VDD | 3.3 V | Do not use 5 V |
| GND | GND | Common ground |
| SCK / BCLK | GPIO5 | I2S bit clock |
| WS / LRCK | GPIO4 | I2S word select |
| SD / DIN | GPIO6 | Microphone data |
| L/R | GND | Selects the left I2S slot |

The two USB connectors have different jobs:

- The CH343 connector remains the firmware flashing and serial-log port.
- The ESP32-S3 native USB connector (GPIO19 D- / GPIO20 D+) is the microphone
  connection. After this firmware starts it enumerates as an audio device, not
  as `USB JTAG/serial debug unit`.

## Project layout

Hardware applications live under `main/boards/<board-id>` so each board owns
its entry point, pin configuration, inputs, audio, display, and other local
drivers. Shared firmware components remain under `components`, while the
internal microphone interface used by the board applications lives under
`main/include`.

To add another board:

1. Add its option to the `AGENTMOTE_BOARD` choice in `main/Kconfig.projbuild`.
2. Create `main/boards/<board-id>` with an `app_main` implementation and the
   board's drivers.
3. Add a `board.cmake` in that directory listing the board's source files, and
   map the new Kconfig option to the directory in `main/CMakeLists.txt`.
4. Add any required ESP-IDF or managed-component dependencies to the central
   component registration and `main/idf_component.yml`.
5. Add a board-specific `sdkconfig.<board-id>.defaults` when the hardware needs
   settings beyond `sdkconfig.defaults`.

## Build

The validated SDK is ESP-IDF v5.5.2. In this environment, `load_idf` is a zsh
alias for loading the SDK. An explicit equivalent is shown below because zsh
aliases do not expand in every non-interactive shell:

```sh
export IDF_PATH="$HOME/esp/esp-idf"
export PATH="$HOME/.espressif/python_env/idf5.5_py3.13_env/bin:$PATH"
source "$IDF_PATH/export.sh"
idf.py set-target esp32s3
idf.py build
```

The default build remains the four-button breadboard. Build StickS3 into a
separate directory so the two configurations never overwrite one another:

```sh
idf.py -B build-sticks3 \
  -D SDKCONFIG=/tmp/agentmote-sdkconfig-sticks3 \
  -D 'SDKCONFIG_DEFAULTS=sdkconfig.defaults;sdkconfig.sticks3.defaults' \
  build
```

Flash StickS3 through its USB-C connector:

```sh
idf.py -B build-sticks3 -p /dev/cu.usbmodemXXXX flash monitor
```

For the breadboard, flash and monitor through the CH343 connector:

```sh
idf.py -p /dev/cu.wchusbserialXXXX flash monitor
```

Exit the monitor with `Ctrl+]`. The exact serial device name varies by board
and USB mode.

## Pair and use

1. Flash the firmware and reboot the board.
2. Connect the native USB port to the Mac.
3. In macOS Sound settings or ChatGPT, select `AgentMote Microphone` or
   `StickS3 Codex Microphone` as input.
4. Open macOS System Settings > Bluetooth and connect to `Codex Micro`.
5. Open the ChatGPT desktop application with Codex Micro support.
6. Hold GPIO47 while speaking and release it to stop.
7. Press GPIO0 to send the transcribed composer content.
8. Press GPIO39 to approve a request, or hold GPIO40 for 700 ms and release to
   decline it.

The USB audio stream is continuously available when a host opens it. The Mic
button only sends the Codex push-to-talk event; it does not start or stop I2S.
If USB is disconnected, the BLE remote continues to work with another selected
microphone. Closed-lid operation also requires the Mac itself to remain awake.

Changing the HID descriptor or identity can leave stale metadata in the host's
Bluetooth cache. If pairing stops working, forget `Codex Micro` on the host,
erase the board's stored bond, reboot, and pair again:

```sh
idf.py -p /dev/cu.wchusbserialXXXX erase-flash flash monitor
```

## Windows

The firmware uses standard BLE HID over GATT, so Windows can pair with and
enumerate the device. Codex command handling still depends on a compatible
desktop host application. This project does not install a Windows bridge or
provide ASR, so BLE enumeration alone does not guarantee that the host will
react to the Mic action on Windows.

The USB descriptor is configured specifically for macOS compatibility. Windows
USB audio behavior is not part of the current validation target.

## Firmware API

Codex control is split into three layers:

1. Board code maps GPIO, gestures, encoders, or sensors to semantic actions.
2. `components/codex_control` owns action mapping, JSON-RPC, HID framing, and
   compatibility identity without exposing ESP-IDF types in its public API.
3. `components/codex_transport_ble_espidf` owns ESP-IDF BLE pairing,
   advertising, reconnect, and report delivery.

An ESP-IDF board creates the controller, starts the supplied transport, and
sends semantic actions:

```c
#include "codex_ble_espidf.h"
#include "codex_control.h"

codex_control_t *control = NULL;
codex_control_config_t config = {
    .firmware_version = "1.0.0-my-device",
    .profile_index = 0,
    .layer_index = 1,
};

codex_control_create(&config, &control);
codex_ble_espidf_start(control);
codex_control_set_battery(control, 95, false);
codex_control_send_action(control, CODEX_ACTION_MIC, CODEX_ACTION_PRESS);
codex_control_send_action(control, CODEX_ACTION_MIC, CODEX_ACTION_RELEASE);
codex_control_send_action(control, CODEX_ACTION_FORK, CODEX_ACTION_PRESS);
codex_control_send_agent(control, 2, CODEX_ACTION_PRESS);
codex_control_send_joystick(control, 0.75f, 1.0f);
```

For another ESP-IDF board, reuse both components and change only its board
configuration and input-to-action mapping. A different MCU platform implements
the callbacks in `codex_control_transport.h` once for that platform; its board
applications still use only `codex_control.h`. The raw-key function is an
explicitly unstable escape hatch and should not appear in normal board code.

The portable core can be built and tested without the ESP-IDF build system:

```sh
cmake -S components/codex_control -B /tmp/codex-control \
  -DCODEX_CONTROL_CJSON_DIR=/path/to/cJSON
cmake --build /tmp/codex-control
ctest --test-dir /tmp/codex-control --output-on-failure

cmake -S components/two_button_input -B /tmp/two-button-input
cmake --build /tmp/two-button-input
ctest --test-dir /tmp/two-button-input --output-on-failure
```

The wire identity and protocol remain private compatibility behavior and may
change in a future ChatGPT desktop release. Physical-device validation should
include pairing from a clean host state, press/release delivery, disconnect
recovery, and host RPC responses.