3 people 95160a304d feat(tools): add model-agnostic computer use (computer_use + browser tools) ...

Adds two opt-in agent tools for controlling a computer:
- computer_use: pixel-based (screenshot + mouse/keyboard) via a desktop
  (pyautogui) or browser (playwright) backend.
- browser: DOM/accessibility-based web automation (act by element ref),
  reliable across ANY tool-calling model, not just vision/CU-trained ones.

Core enabler in the runner: a tool may return image content blocks, which
are split out and delivered to the model as a follow-up user message
(_split_tool_result_media), so screenshots reach any vision provider
(e.g. via OpenRouter/openai-compat) without provider-specific code.

Both tools are OFF by default (tools.computerUse.enable / tools.browser.enable),
are not exposed to subagents, and the browser tool supports an allowed_domains
allowlist. Heavy deps (pyautogui/pillow/playwright) are an optional [computer-use] extra.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-08-09 01:12:13 +09:00

..

agent

fix(agent): stop session-owned exec processes

2026-08-08 23:20:59 +08:00

bus

feat(sdk): add host integration extension points

2026-07-28 15:30:28 +08:00

channels

fix(channels): preserve global progress defaults

2026-08-07 17:19:14 +08:00

cli

fix: modernize dependency recovery guidance (#5282)

2026-08-07 13:58:13 +08:00

cli_apps

refactor: remove verified dead code

2026-08-08 21:10:34 +08:00

command

fix(commands): reject invalid slash commands

2026-08-04 17:11:44 +08:00

config

refactor(webui): improve visual consistency (#5249)

2026-08-05 13:24:45 +08:00

cron

test(cron): cover invalid schedule expressions

2026-08-03 16:20:22 +08:00

e2e

feat(tools): add model-agnostic computer use (computer_use + browser tools)

2026-08-09 01:12:13 +09:00

gateway

feat(webui): add guided setup flows

2026-07-13 13:11:46 +08:00

pairing

fix(pairing): keep approvals across transient store read failures

2026-07-30 19:02:37 +08:00

providers

fix: modernize dependency recovery guidance (#5282)

2026-08-07 13:58:13 +08:00

security

fix(image): delegate DNS to explicit proxy

2026-07-27 10:06:19 +08:00

session

fix(webui): complete temporary chat mode

2026-08-08 23:20:59 +08:00

tools

feat(tools): add model-agnostic computer use (computer_use + browser tools)

2026-08-09 01:12:13 +09:00

triggers

fix(webui): show actual local trigger messages (#5228)

2026-08-03 16:43:01 +08:00

utils

fix(webui): complete temporary chat mode

2026-08-08 23:20:59 +08:00

webui

fix: modernize dependency recovery guidance (#5282)

2026-08-07 13:58:13 +08:00

test_api_attachment.py

refactor: enforce BasedPyright strict type checking (#5158)

2026-07-29 21:37:11 +08:00

test_api_stream.py

fix: require api auth in server factory

2026-07-08 12:16:12 +08:00

test_build_status.py

fix(status): correct context percentage calculation and sync consolidator

2026-04-16 20:30:39 +08:00

test_context_documents.py

fix(agent): read document attachments on demand (#5122)

2026-07-28 13:33:06 +08:00

test_docker.sh

fix: keep onboard cli defaults unchanged

2026-06-22 13:04:05 +08:00

test_document_parsing.py

refactor: remove verified dead code

2026-08-08 21:10:34 +08:00

test_file_tool_toggle.py

fix: preserve file tool toggle for subagents

2026-06-15 02:55:18 +08:00

test_nanobot_facade.py

feat(config): add actionable startup diagnostics and WebUI recovery (#5110)

2026-07-28 18:52:05 +08:00

test_openai_api.py

refactor(core): remove redundant runtime scaffolding (#5127)

2026-07-28 11:07:58 +08:00

test_package_version.py

test: speed up CI and harden the suite

2026-07-15 00:18:37 +08:00

test_tool_contextvars.py

refactor(agent): capture subagent runtime before spawn

2026-07-10 17:54:34 +08:00

test_truncate_text_shadowing.py

fix(agent): avoid truncate_text name shadowing

2026-04-10 17:36:31 +08:00