What problem are you trying to solve?

This is a meta tracking entry for performance issues mixing multiple canvas/video types:

Goal:

• Easier and fast image transfers between various framebuffer types (Canvas2D, WebGL, WebGPU, VIDEO) while preserving colorspace (including HDR)

Example Use Cases:

• Browser-based streaming apps (casting relqays, streaming over LAN for remote play, etc)
• Offscreen-rendered Canvas2D Text overlays on WebGL/WebGPU
• Fast compositing (e.g. using any Canvas2D/WebGL/WebGPU images on any Canvas2D/WebGL/WebGPU)

Current Example Problems Today:

• Not all combinations are easy/fast (e.g. Grabbing WebGPU frames seems to requires pixel by pixel copy)
• Poor Performance for multiple reasons (inefficient copies, colorspace conversions, GPU-CPU-GPU, etc)
• Unwanted colorspace conversions (e.g. byte->float16) due to cross-API limitations;
• Many methods lose colorspace (see Adding support for High Dynamic Range (HDR) imagery to HTML Canvas: a baseline proposal #9461 )
  Example: Loss of HDR (e.g. copying Canvas2D HDR to WebGPU HDR doesn't work yet, or trying to stream a HDR framebuffer over WebRTC HDR)
• Memory transfer impact (GPU memory -> CPU memory -> GPU memory)
• Power inefficency, lots of battery power used with some current APIs (compared to native APIs available by graphics drivers etc)
• Feature detection is a bit inconsistent (see Feature detection for HDR canvas gpuweb/gpuweb#4828 as example)

More Info

Long-term Potential Decision Points

• EDIT: ImageBitmap API supports all canvas types (TBD: Need to test colorspace compatibility issues in the new boom of HDR)
• EDIT: copyExternalImageToTexture() can copy any compatible (image/all canvas) to a WebGPU texture. This likely is probably a GPU-GPU transfer.
• Whether to piggyback on existing browser Capture API (.captureStream) or if we need to expand beyond this.
• Whether or not to have a single API for this; or have separate high-performance APIs;
• And if APIs remain separate, whether to provide higher-performance APIs on a per-canvas format for bidirectional image transfers;
• Deciding how to plan ahead on this due to the explosion of colorspaces (HDR) to prevent architecturing ourselves into a corner;
• Deciding whether to skip optimizing legacy APIs may be fine (e.g. ignore WebGPU, focus only on Canvas2D/WebGL/VIDEO)
• Evaluating potential HTML API approaches (what should be HTML API and what should be browser engine optimizations);
• Evaluating potential engine optimization approaches (where accomodable by existing HTML APIs)

Example Canvas2D workflows

• transferToImageBitmap
• getImageData
• getImageDataAsync (new API, see CanvasRenderingContext2D: Add getImageDataAsync method #10855)
• putImageData

Example non-Canvas2D workflows

• WebGL/WebGPU createTexture()
• WebRTC uses captureStream()

Browser capture APIs

The most commonalized method is .captureStream() that can be used on any of these elements but there are some severe constraints:

• Loosely coupled (e.g. not synchronized to the canvas frame rate)
• Must specify a fixed framerate (e.g. 30 frames per second);
• Often not colorspace synchronized
• Design for video streaming rather than other use cases (e.g. compositing use cases or framebuffer-modification use cases).
• Often still has GPU-CPU-GPU transfers if you need to do certain kinds of compositing

Role of operating systems:

More OSes are providing more methods of centralized APIs to get any framebuffer types, e.g. Zoom casting of any graphics window (non-HDCP video window, 3D graphics app, 2D app, etc). With the addition of simpler OS-level APIs due to the casting culture, browser-level APIs are still a zoo in this territory.

Role of HTML APIs:

One possible idea is towards a standardized name such as getImageDataAsync (#10855) to be commonalized among all canvas and canvas-style framebuffer APIs (WebGL, WebGPU, Canvas2D, non-HDCP video)

Some roles is played by HTML APIs and some roles may be played by browser engine optimization, but I post it here because there's an HTML API role to play (e.g. theoretical getImageDataAsync compatible with all framebuffer types, if a decision is made to follow that path in the future)

Role of browser engine optimization:

Example theoretical browser engine optimization that keeps it all on GPU to avoid bus transfers to CPU: When converting bitmaps to textures and vice-versa, such as low-level zero-copy memory transfers. Sometimes you can use ultra high speed GPU-GPU memory copying, then lock that GPU memory and memory-map to CPU RAM, without needing to halt/stop ongoing rendering (since the GPU is not using that copy anymore).

In other words, memory mapped clone of frame to CPU RAM as a zero-copy transfer. In theory, getImageData/getImageDataAsync + createTexture() could be "implemented" this way to eliminate memory transfers from GPU to CPU and back to GPU.

Please note this is a theoretical optimization of existing "copy GPU to CPU" APIs, but this does not solve the simplification portion (which may actually reduce browser engine code size long-term, given some OS-level optimizations going on long-term).

Dream Goal:

A simple unified API path that does a high performance GPU-GPU memory copy (that preserves colorspace) between any framebuffer type: Canvas2D, WebGPU, VIDEO, and maybe WebGL (legacy), including the option of synchronized frame rates (. All fast enough for real-time compositing.

Realistic Goal:

Something better than the current complicated zoo that is getting more complicated thanks to introduction of HDR (e.g. Safari Tech Preview 215 supports HDR in WebGPU)