DSH Computer Use
Text-first browser and background macOS control for DSH.
Target the right process and window without taking over the user's pointer.
简体中文 ·
Install
Homebrew
brew tap zrui-c/tap
brew trust zrui-c/tap
brew install --cask dsh-computer-use
open -a "DSH Computer Use"DMG
- Download the latest
DSH-Computer-Use-*-universal.dmgfrom Releases. - Drag DSH Computer Use into Applications and open it.
With either method, authorize Accessibility and Screen Recording, select Install under DSH Plugin, then restart the running DSH Host.
The Homebrew Cask and official DMG install the same Universal 2, Developer ID signed, Apple-notarized app. Users do not need Xcode, Swift, or this source checkout. DSH and Google Chrome must already be installed.
What it does
| Surface | Perception | Input |
|---|---|---|
| Chromium | Playwright, CDP accessibility/DOM, frames, tabs, optional OCR | Ref-pinned navigation, pointer, keyboard, forms, scroll, drag, upload |
| macOS | Accessibility tree first, Vision OCR for semantic gaps, independent window capture | AX actions, targeted SkyLight/CoreGraphics, global HID only without a target |
Every action returns a fresh bounded text observation. The model works with roles, names, values, state, geometry, and stable snapshot refs instead of assuming it can inspect screenshots. UI and OCR strings are explicitly untrusted data.
Background macOS control
- Carries PID, WindowServer window ID, AX window frame, and element identity through every action.
- Prefers semantic AX actions before coordinate input.
- Routes supported pointer and keyboard events directly to the target process/window.
- Uses a click-through software cursor; targeted actions do not move the physical pointer.
- Keeps post-action observation pinned to the previous target, even while another app stays active.
- Falls back to public CoreGraphics or fails closed when a private capability is unavailable.
Platform boundary
ScreenCaptureKit is public API. The optional background input path dynamically loads private SkyLight symbols and is intended for Developer ID distribution, not the Mac App Store. Private APIs are unsupported by Apple and can change between macOS releases.
On macOS 26, Stage Manager may expose a shelved window only as a small WindowServer thumbnail. Constructing a capture filter for that representation can abort inside SkyLight. DSH Computer Use detects the geometry mismatch first, preserves AX observation, returns an explicit warning, and never stretches a thumbnail into a fake full-window screenshot.
DSH integration
The embedded package declares a DSH bundle in package.json. The setup center runs the official equivalent of:
dsh plugin --profile web add --save-exact file:/path/to/DSH\ Computer\ Use.app/Contents/Resources/Plugincordis.patch.yml installs the Host runtime and registers computer_observe / computer_action in DSH's global tool layer, inherited by every agent preset. No manual edits to user profile YAML or preset copies are required. The setup center detects an older dependency-only installation and repairs the missing bundle registration. A running DSH Host must be restarted after install, repair, or upgrade.
Build from source
Requirements: macOS 14+, Xcode/Swift 5.9+, Node.js 22+, pnpm 11+, DSH, and Google Chrome.
pnpm install
pnpm run typecheck
pnpm run test
pnpm run test:native
pnpm run buildpnpm run build creates:
native/macos-helper/dist/DSH Computer Use.appThe default build is Universal 2. For faster local iteration:
COMPUTER_USE_ARCHS=arm64 pnpm run buildCreate a local drag-to-Applications DMG:
pnpm run package:dmgPublic releases require a Developer ID Application identity and notarization. See documentation/distribution.md for the exact local and GitHub Actions flows.
Tool contract
computer_observe returns interactive, full, or changes snapshots for browser and desktop, with optional query filtering and auto | always | never OCR.
computer_action performs exactly one browser or desktop action and returns the post-action semantic state. Ref and coordinate actions require the latest snapshot_id; stale targets fail closed and require another observation. File uploads are fenced to the DSH session workspace.
Project
Community
- GitHub Discussions — ask questions, share usage, report ideas
- DeepSeek Harness Discord — the wider DSH ecosystem
- Star the repo if DSH Computer Use saves your pointer 🖱️
Licensed under Apache-2.0. This independent project is not endorsed by Apple. “DeepSeek” and related marks belong to their respective owners; the name is used only to describe compatibility with DeepSeek Harness/DSH.
