Engineering journal
Giving an AI Tool Images Over MCP Without Losing the Canonical Observation
Cogniform adds optional, bounded PNG content to MCP observations while keeping exact values, stable identities and causal metadata in the canonical resource.
About 4 min read
An image is an efficient way for a person or an AI tool to inspect a scene. It is a poor substitute for the exact observation that explains what the image represents. Cogniform's CF083 change gives MCP callers both: an optional diagnostic PNG and the existing canonical observation resource.
The feature merged in PR #84. It changes the presentation of cogniform.observe_scene, not the authority to modify a world. Cogniform remains an early source implementation with unreleased packages.
Keep one authoritative observation
The canonical COGOBS01 resource carries exact numeric data, stable entity identities and causal metadata. A diagnostic PNG transforms that data into something directly inspectable. Depth becomes inverted normalized grayscale, world normals become display colors, and entity IDs receive deterministic colors with a transparent background.
Those transformations answer practical questions about appearance, occlusion, orientation and segmentation. They lose information. A depth image does not preserve the exact numeric depth payload, and distinct entity identities can receive the same display color. Exact reasoning still needs the canonical resource.
The CF083 decision record (opens in a new tab) makes the relationship explicit: presentation is optional, and diagnostic output does not replace the observation contract.
Opt in without changing the omission path
A caller adds presentation: "png" to the existing observation request for color, depth, normal or entity-ID observations. The result includes text, a base64 PNG image block and the canonical resource link. Structured output also describes the image's MIME type, decoded size and diagnostic status.
When the field is omitted, the previous text-and-resource-link sequence and structured result remain unchanged. There is no extra tool to discover. Visibility observations have no PNG projection and reject before lazy service creation. Unsupported presentation values reject as well.
This preserves a simple compatibility boundary: an existing client does not need to accept images merely because the server can now return them.
Bound completion, not just dimensions
Each complete encoded PNG is capped at 1,048,576 bytes, or 1 MiB. That is independent of the canonical envelope's 4 MiB bound. Base64 makes the wire representation larger, so the decoded-image cap is not a promise that the entire tool response fits in 1 MiB. The adapter also retains its separate output-line limit.
The implementation checks dimensions, item counts, arithmetic and allocation. A pure borrowed-input encoder in the observation module is shared with the existing CLI graphics commands. The source validation reports byte-identical pre/post CLI bundles, which helps prevent the MCP view from drifting away from CLI diagnostics.
Publish only a complete result
The sequence matters. Prepare the canonical resource first, then the optional PNG, then the complete tool result. Replace the retained resource only after all three succeed.
If image preparation fails, the previous retained resource survives. This does not create a resource history: the protocol retains only the latest successful observation. Failure handling also remains specific to the error; preserving an old resource does not imply every failing child is safe to reuse. The pinned MCP protocol (opens in a new tab) defines those boundaries.
Useful images with explicit limits
The source validation covers the four supported image kinds, omission, bounds, base64, transport capacity and prior-resource preservation. Controlled tests exercise both accepted MCP lifecycles and real rendering. This portfolio reviewed that public evidence; it did not run a new GPU test or publish a new observation capture.
PNG work is cooperatively bounded, not preemptible. Diagnostic images may contain sensitive scene information. Cross-adapter byte identity is not a conformance guarantee. There is no new remote listener, persistence, model access or mutation authority.
The useful pattern is to make inspection easier while keeping the exact evidence reachable. Read the short engineering note, then follow why observations need world revisions for the causal context behind the image.