Module computer_use 

HTTP server implementing the Anthropic Computer Use spec for AI agent desktop control. HTTP server implementing the Anthropic Computer Use specification.

Enabled by setting the PIXELFLUX_CU environment variable to a bind: comma-separated entries, each a port, served on the loopback addresses only, or host:port, the address to serve on (0.0.0.0:5000,[::]:5000 serves every interface). The server handles POST /computer-use requests for screenshots, mouse/keyboard injection, scrolling, and cursor position queries. Actions run against a [CuBackend], resolved per request: the Wayland compositor owned by this process when one is registered, otherwise the X server named by DISPLAY (XTEST injection on a private connection, no active capture required).

A caller drives the session with its owner’s full authority, and a loopback address is open to every account on the host, so the server does not start without a token (PIXELFLUX_CU_TOKEN, or the caller’s own through [start_cu_server]) and every request has to carry it as Authorization: Bearer <token>.

The same server also exposes the built-in MP4 recorder at /record_start, /record_stop, and /record_status, so a headless script can drive a session and record it over plain HTTP. A recording the caller names is a file name inside PIXELFLUX_RECORD_DIR, the directory the operator gives callers, and nowhere else.

Structs

CuWaylandBackend

Wayland implementation: every primitive is a ThreadCommand on the compositor’s calloop channel, so injection and readback serialize naturally with rendering and encoding.

Enums

CuButton

Semantic mouse-button vocabulary of the action layer; each backend maps it to its native codes (evdev BTN_ on Wayland, X core buttons 1/2/3 on X11).

Traits

CuBackend

The desktop-side primitives one Computer Use action decomposes into. Keycodes are X/xkb keycodes (evdev + 8) — the numbering both the smithay seat and XTEST consume — and coordinates are framebuffer/root pixels, already clamped by the action layer.

Functions

encode_png_rgba

Turn a raw framebuffer into a PNG the Computer Use agent can actually look at.

register_wayland_backend

Make the Wayland compositor the preferred CU backend: once a live calloop sender is registered, every subsequent request routes to it instead of an X11 connection.

run_cu_server

Expose the captured desktop to an AI agent over HTTP, so a Computer Use client can drive the session much as a human viewer would.

set_app_wayland_display

Set (or clear, with None/empty) the app compositor socket for CU text injection and for relative pointer motion into a nested KWin session.

spawn_cu_from_env

Start the CU server if PIXELFLUX_CU names a bind (the standalone fallback; a selkies-managed session passes the setting through start_cu_server).

start_cu_server

Start the CU server on bind, comma-separated entries: a bare port listens on the loopback addresses only, host:port names the address to listen on (0.0.0.0:port,[::]:port accepts every interface), requiring token, else PIXELFLUX_CU_TOKEN, of every request. Guarded so that exactly one server binds per process no matter how many call sites (module import, Wayland compositor init, the selkies setting) race to spawn it; backend selection stays per-request, so a server bound at import serves a compositor that only starts later.