rokid-glass-browser

KO | EN

gitlite — search

#rag #javascript #ui #gemini #vue #github #http #windows #shell #privacy

rokid-glass-browser

(★ 9)

Standalone Rokid Glass Browser

File Explorer

ZIP Download

- - README.md
- agent-console.png
- agent.png
- glasses-hud.png
- google-layout.png
- rv101-input-research.md
- text-keyboard.png
- transparent-video.png
- url-keyboard.png
- voice-input.png
- waiting.png
- web-remote.png
- - .gitignore
  - build.gradle.kts
  - gradle.properties
  - settings.gradle.kts
- - icon.png
  - web_remote.html
- - isolated-runner-design.md
  - media-tools-result.md
  - NODE_RUNTIME.md
  - run-node-runtime-decision.md
- - agent_files.dart
  - agent_settings.dart
  - agent_vision.dart
  - browser_agent.dart
  - browser_screen.dart
  - main.dart
  - url_keyboard.dart
  - voice_asr.dart
  - web_remote_server.dart
- - web_remote_server_test.dart
  - widget_test.dart
- .gitignore
- .metadata
- analysis_options.yaml
- NODE_RUNTIME.md
- pubspec.lock
- pubspec.yaml
- README.md
- rokid-glass-browser-1.31.1.apk
.gitignore
LICENSE
README.md

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

A standalone web browser for **Rokid AI Glasses (RV101, Android 12)** that you drive from any phone's web browser over Wi‑Fi — no companion app to install, no Bluetooth pairing, no cloud.

The glasses run the browser **and** host a tiny web page. Open that page on your phone (same Wi‑Fi) and you get a live view of what the glasses show, a touchpad, and all the controls. The glasses' own touchpad and temple button are fully mapped too, so you can browse hands‑free once a page is open.

> Inspired by [Inplov/rokid-browser](https://github.com/Inplov/rokid-browser) (Bluetooth‑paired phone app). This project is a from‑scratch rewrite around a different idea — the glasses serve the remote — and shares no code with it. The hardware‑button technique was learned from [ksuzukigh/rokid-zoom-in-camera](https://github.com/ksuzukigh/rokid-zoom-in-camera). Thanks to both authors.

![Web remote](https://raw.githubusercontent.com/xnohat/rokid-glass-browser/main/docs/web-remote.png)
![URL keyboard on the glasses](https://raw.githubusercontent.com/xnohat/rokid-glass-browser/main/docs/url-keyboard.png)
![Text keyboard for page inputs](https://raw.githubusercontent.com/xnohat/rokid-glass-browser/main/docs/text-keyboard.png)
![Voice input](https://raw.githubusercontent.com/xnohat/rokid-glass-browser/main/docs/voice-input.png)
![AI agent result](https://raw.githubusercontent.com/xnohat/rokid-glass-browser/main/docs/agent.png)
![Floating agent console](https://raw.githubusercontent.com/xnohat/rokid-glass-browser/main/docs/agent-console.png)

## What you get

**On the phone (any browser, iOS or Android)**
- Live stream of the glasses display (JPEG, ~5 fps, ACK‑paced) — collapsible so it doesn't eat the screen
- Address bar with search, ✕ clear
- Touchpad: 1 finger = move cursor, tap = click, hold‑then‑drag = drag, 2 fingers drag = scroll, pinch = zoom
- Arrow keys ▲▼◀▶ around the touchpad, Back / Forward / Reload, Enter / Backspace, text input
- Display modes (normal / transparent / wireframe), dimming, text zoom, HUD toggle
- **Exit browser on glasses** button
- **Agent log** (expand under the command box): every run's steps — tool, arguments, duration, result — so prompts and tools can be tuned from real data; Copy JSON to share it
- **AI Assistant settings** (expand): **Conversation memory** on/off (remembers the current app session so you can chat back‑and‑forth like an assistant; “Clear conversation now” to forget), **Speak the reply** on/off + a **voice** picker (the assistant reads only its final answer aloud via Gemini TTS — never the tool steps), and an editable **persona** system prompt (default: *Ani, a 20‑year‑old witty, sweet female assistant*). The persona is layered on top of the browser‑automation prompt — it never replaces it, so tool use stays reliable
- **History** section (expand at the bottom): search, open, delete one, or clear all
- Auto‑reconnect after reload or a short network drop; one controller at a time; revoke/stop from the glasses

**On the glasses**
- **Floating agent console** (bottom-right, shows while the agent runs): what you said (🗣), the model's thinking (💭), each tool/command it runs (⚙︎) and the final result (✓), auto-scrolling to the newest line
- **AI agent (hold the temple button)**: hold → beep + the console shows **LISTENING**, say what you want, press the button once → the command is transcribed and an agent carries it out with browser + app tools (navigate / read_page / click / type / press_enter / scroll / back / forward / reload / app_action). **Hold again** (while listening or running) cancels and re‑listens; **double‑tap** the touchpad cancels the agent. Works for one‑shot commands (“refresh”, “scroll down a bit”) and multi‑step ones (“open YouTube and find videos about palantir ontology”). Same agent can be driven from the web remote's command box
- All on-glasses UI is in English
- **Voice input (🎤 on both keyboards)**: tap once → beep + “ĐANG NGHE…”, speak, tap ⏹ → text is recognised by Gemini (default **gemini-3.8-flash** (cheapest); pick any model from the web remote after *Fetch models*) and typed into the field. Needs a Gemini API key saved from the web remote (Settings → *Gemini API key*, stored on the glasses only). The RV101 has no Google Speech, so sites’ own voice search cannot work; this replaces it
- **Microphone for web pages** (YouTube / Google voice search works; permission asked once on first launch)
- Page content is laid out **below** the address bar (no more fixed headers hidden under it — e.g. the YouTube search button)
- **Wi‑Fi auto-on** when the app opens (also persists Rokid's own Wi‑Fi preference so it survives reboot — technique from [rokid-wifi-on](https://github.com/ksuzukigh/rokid-wifi-on))
- **URL keyboard** (click the address bar): QWERTY + digits + `: / ? . # % @`, `http://` `https://` `www.` `.com`, ✕ clear, GO — big keys for the 2‑axis cursor. Autocomplete from your history and popular services (root domains first, deep paths after), ▲▼ to pick, ✕ to forget an entry
- **Text keyboard**: click any input on a page and a compact keyboard slides up (letters, digits, symbols, ⇧, ␣, ⌫, **✕ Xóa ô** clears the field, ENTER submits). Clicking the page hides it, like a phone
- **Mouse mode double-tap** on the touchpad = double-click at the cursor
- Toolbar next to the address bar (click with the cursor): ◀ ▶ Back/Forward, ‹ › ˄ ˅ arrow keys to the page, ■ stop, ↻ reload, ⏻ exit
- Two modes, toggled with the temple button: **MOUSE** (swipe moves the cursor, tap clicks) and **SCROLL** (swipe scrolls / jumps between elements, tap activates)
- Smooth cursor glide with acceleration on repeated swipes; cursor stays visible in mouse mode
- Element‑jump starts from the first element currently on screen, not the top of the page

## Controls on the glasses / Điều khiển trên kính

**Touchpad & temple button** — EN / VI

| Input · Thao tác | Mouse mode · Chế độ chuột | Scroll mode · Chế độ cuộn |
|---|---|---|
| Swipe forward / back · Vuốt tới / lui | Move cursor (axis set by button) · Di chuột (trục do nút chọn) | Scroll ⅓ screen / jump element · Cuộn ⅓ màn hình / nhảy phần tử |
| Tap 1 finger · Chạm 1 ngón | Click at cursor · Bấm tại con trỏ | Activate focused element · Kích hoạt phần tử |
| Double‑tap 1 finger · Chạm 2 lần 1 ngón | Double‑click — **or cancel the AI agent if it is listening / running** · Nhấp đúp — **hoặc huỷ AI agent nếu đang nghe / đang chạy** | Back · Quay lại |
| Temple button ×1 · Nút thái dương ×1 | Flip cursor axis ↔ / ↕ · Đổi trục chuột ↔ / ↕ | (shows current mode · hiện chế độ) |
| Temple button ×2 · Nút ×2 | Switch MOUSE ↔ SCROLL · Đổi CHUỘT ↔ CUỘN | same · như trái |
| Temple button hold · Giữ nút | **AI agent: listen** (see below) · **AI agent: lắng nghe** (xem dưới) | same · như trái |

**AI agent — hold‑to‑talk** · **AI agent — giữ để nói**

| Action · Hành động | Result · Kết quả |
|---|---|
| **Hold** the temple button · **Giữ** nút thái dương | Beep + the agent console pops up in **LISTENING** state; speak your command · Kêu bíp + cửa sổ agent hiện trạng thái **LISTENING**; nói lệnh của bạn |
| **Press once** while listening · **Bấm 1 lần** khi đang nghe | Stop recording → transcribe → the agent runs the command · Dừng ghi âm → nhận dạng → agent thực thi lệnh |
| **Hold again** while listening **or** while the agent is running · **Giữ lại** khi đang nghe **hoặc** khi agent đang chạy | Cancel the current run and **start listening again** (console clears, then LISTENING…) · Huỷ lượt hiện tại và **nghe lại từ đầu** (cửa sổ xoá rồi hiện LISTENING…) |
| **Double‑tap** the touchpad while listening **or** running · **Chạm 2 lần** touchpad khi đang nghe **hoặc** đang chạy | Cancel the AI agent entirely · Huỷ hẳn AI agent |

**Conversational assistant · Trợ lý hội thoại:** with *Conversation memory* on (default), the agent remembers the whole app session, so you can talk to it back‑and‑forth. Turn on *Speak the reply* to hear its answer (Gemini TTS voice), and edit its *persona* — all in the web remote's **AI Assistant** settings. · Khi bật *Conversation memory* (mặc định), agent nhớ cả phiên làm việc để bạn đối thoại qua lại; bật *Speak the reply* để nghe câu trả lời (giọng Gemini TTS) và sửa *persona* — tất cả trong mục **AI Assistant** của web remote.

**Multimodal (see / watch / listen) · Đa phương thức:** the agent can look at what's on screen (`see_page`), understand the playing video (`watch_video` — YouTube URLs go straight to Gemini's video understanding), and hear audio (`listen_audio`), all via Gemini. Ask “what's in this picture / what is this video about / what did they just say”. · Agent có thể xem ảnh trên màn hình, hiểu video đang phát (YouTube gửi thẳng URL cho Gemini) và nghe âm thanh — hỏi “ảnh này là gì / video này nói gì / vừa nói gì vậy”.

The agent console (bottom‑right) shows what you said (🗣), the model's thinking (💭), each tool it runs (⚙︎) and the result (✓). Exit the browser from the **⏻** icon on the toolbar, the web remote, or the start panel — not from the temple button (that is the agent now). · Cửa sổ agent (góc dưới phải) hiện câu bạn nói (🗣), suy nghĩ của model (💭), từng công cụ chạy (⚙︎) và kết quả (✓). Thoát trình duyệt bằng biểu tượng **⏻** trên thanh công cụ, web remote, hoặc bảng khởi động — không dùng nút thái dương nữa (giờ dành cho agent).

While the browser is in the foreground it asks the Rokid system to disable the button's photo/video actions, and restores them when you leave — same mechanism as rokid‑zoom‑in‑camera. Two‑finger gestures are owned by the Rokid system (volume, AI assistant) and cannot be remapped by an app; see `docs/gesture-lab/README.md` for the full measured gesture table. · Khi trình duyệt đang mở, app yêu cầu hệ thống Rokid tắt chức năng chụp ảnh/quay phim của nút và khôi phục khi thoát. Cử chỉ 2 ngón do hệ thống Rokid chiếm (âm lượng, trợ lý AI) nên app không map lại được.

## Agent workspace & tools / Workspace và công cụ Agent

The agent has a private workspace inside the app. Paths cannot escape this directory. · Agent có workspace riêng bên trong app; đường dẫn không thể thoát ra ngoài.

- **Files · Tệp:** list/read/write/delete, HTTPS download (25 MB cap), and Gemini understanding for image/audio/video files.
- **Python:** embedded **Python 3.13.5** in a disposable `:python` process; `run_python` has timeout/output bounds. `install_python_package` installs universal pure-Python wheels only (`*-none-any.whl`) into `.python`; Dart/Android TLS downloads the wheel and pip installs locally with `--no-index --no-deps --only-binary=:all:`. Verified with `six==1.17.0`.
- **Shell:** allowlisted Android toybox commands in the workspace (`pwd`, `ls`, `cat`, `cp`, `mv`, `grep`, `sed`, `find`, `head`, `tail`, etc.); no arbitrary shell operators or paths outside the workspace.
- **Git:** local repositories via Eclipse JGit: init/status/add/commit/log. No remote credentials or push/pull. Verified on-device through a clean commit cycle.
- **Camera & multimodal · Camera và đa phương thức:** `see_page`, `watch_video` (YouTube URL goes directly to Gemini), `see_camera`, `watch_camera`, and microphone/ambient listening.
- **History · Lịch sử:** current-session agent history stays in the fixed console, can be reviewed with touchpad swipes, and is available in full on the web remote with per-turn delete and clear-all.

**Not shipped · Không ship:** Node/npm (YodaOS SELinux blocks Node/libuv `/proc/self/cwd`), FFmpeg/FFprobe executables (YodaOS denies executing the packaged PIE binaries), and ImageMagick/Pillow native imaging (native module load blocked). Details: `docs/node-same-apk-rv101-result.md`, `docs/media-tools-result.md`. No broken tool is advertised.

## Install

Requirements: Rokid RV101 with developer mode (ADB) enabled and the 5‑pin debug cable, or install the APK through Hi Rokid → Toolbox → Glasses app management.

```bash
adb install -r releases/rokid-glass-browser-1.31.1.apk
```

## Use

1. Put the glasses and your phone on the same Wi‑Fi.
2. Open the browser on the glasses → tap the **Web Remote** button (bottom‑right) → **BẮT ĐẦU WEB REMOTE**.
3. The glasses show an address like `http://192.168.1.x:NNNNN`. Open it on your phone. That's it — no code to type.
4. To cut the phone off: **THU HỒI** (revoke) or **DỪNG** (stop) on the glasses.

The port is random per start and the server only listens on the LAN interface; it is not reachable from the Internet. Only one phone can control at a time.

## Build from source

```bash
cd glasses_app
flutter pub get
JAVA_HOME=<JDK 17> flutter build apk --release --target-platform android-arm64
```

Tested with Flutter 3.x / Dart 3.11, JDK 17. `flutter test` covers the local server (auth, origin/host checks, one‑controller, frame ACK/backpressure, reconnect grace, revoke/stop).

## Repository layout

```
glasses_app/   Flutter + Kotlin app that runs on the glasses (browser + web remote server)
docs/          Gesture lab measurements and RV101 input research
releases/      Signed release APKs
```

## Known limits

- Stream is JPEG over WebSocket, capped at 5 fps — fine for navigation, not for watching video on the phone.
- Two‑finger double‑tap on the touchpad is recognised by the firmware only ~40% of the time; it is used only as a Web Remote panel toggle.
- The phone page has been tested in Chromium‑based browsers and iOS Safari for pairing/stream/control; if something misbehaves on your browser, open an issue with the browser name.

## License

MIT

rokid-glass-browser

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

rokid-glass-browser

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README