Graphical sessions

uterm’s session model — one stream, a lease that says who may type, presence for everyone watching — is not specific to a character grid. The graphical stack applies it to a remote desktop: an agent or an operator can capture the screen, click, drag, and type against a target, under the same hijack lease that governs a PTY.

Two ways in

Programmatic control. A GraphicalSession captures an RgbaImage and injects pointer and key events. This is the surface the 7 gui_* MCP tools sit on — gui_screenshot, gui_click, gui_drag, gui_type, gui_key, and the gui_hijack_begin / gui_hijack_release pair that acquires the lease first.

Human relay. A browser can also be bridged straight through to an upstream RFB (VNC) server. Upstream→browser is pumped as raw bytes; browser→upstream passes through an input filter that gates KeyEvent, PointerEvent, and ClientCutText on the same authorization callback the rest of the control plane uses. The handshake and every non-input message pass through untouched.

The filter fails closed: with no injection callback wired, input messages are dropped rather than forwarded. A relay that cannot check permission does not get to guess.

Screenshots without an imaging dependency

encode_rgba_png serialises a raw RGBA8888 framebuffer to PNG using nothing but zlib for the IDAT stream and binascii.crc32 for chunk CRCs. The Go and C# ports carry the same encoder, and they are held to producing byte-identical output — zlib RLE and all. An agent that captures a screen through the Go server and one that captures it through Python get the same bytes, which is what makes a screenshot comparable across a mixed fleet.

Framebuffer dimensions are capped at 8192 per side. A hostile ServerInit announcing a 60000×60000 desktop is refused at the boundary rather than allocating on the strength of a remote number.

Targets are registry entries, not connection strings

A GraphicalTargetDefinition describes a console — protocol (memory or rfb), endpoint, credentials — and lives in a registry rather than in a client’s request. Every read and write is gated by a tenant scope derived from the authenticated principal, and public_copy strips secrets from anything crossing the REST boundary, so a target’s password is never in a response body.

See multi-tenancy for how that scope is derived.

Where it lives

C# is the canonical implementation for the graphical-target registry and the GUI operations; Python and Go are held to it. The Python RFB client is still deferred — the memory protocol backs tests and offline tooling — while the human relay path against a real upstream VNC server is implemented.