Add HTML-in-Canvas APIs - #11588

Wasn't it supposed to be set on Element? (WICG/html-in-canvas#134 (comment))

Yeah, should be elements, not "HTML elements", so SVG and MathML can be drawable.

There are three parts to how making drawable work on any element:

• The algorithms that create and draw snapshots should just talk about elements with the drawable attribute. This is already the case, so per current spec it should work.
• Where drawable is reflected. Instead of putting things on Element we more often use HTMLOrSVGOrMathMLElement, and I think that's probably the right call here too.
• Conformance or requirements for authors. It should really be up to the SVG and MathML specs to say where drawable attributes can appear. I'll file issues on those specs.

Assuming that MathML and SVG agree to allow drawable, which it seems like they will, what do we say here? That is, assuming everything @foolip says above happens, what would this text say? Is it just "All ELements ..."?

Does that mean that if we draw an element on a canvas, then move it back into another container without calling clearElementGeometry, that element will still have the offset for hit testing?

@Kaiido -- No, that's not the intention. We are still dialing in the behavior on this, but our intention is that the canvas transform associated with an element will be cleared when a DOM change occurs that makes the element no longer eligible to be drawn in the canvas.

@szager-chromium can you give me the details of what needs to be spec'd here? Removal steps that look clear the canvas transform and any associated snapshot? Should the same also happen (synchronously) when the drawable attribute is removed?

Need more explication of width and height. Here's some non-conforming text:

These dimensions, when used as the dx and dy parameters to drawElementImage, will cause the image to be drawn at the same size it would be were it placed outside the canvas. when the dx and dy parameters are omitted from a call to drawElementImage, these are the default values that are used.

For the WebGL and WebGPU interfaces, if these dimensions are used as the destination size when drawing to a texture; and the texture is then blitted directly into the canvas (i.e., 1:1 texel-to-canvas-pixel ratio), then the element will appear in the canvas with the same size it would have were it placed outside the canvas.

I think it would be nifty if these definitions could somehow be worked into the existing definition of "natural dimensions", but that's beyond my spec-fu.

will cause the image to be drawn at the same size it would be were it placed outside the canvas

Should note (if not noted already somewhere) whether this is assuming the default canvas sizing (1 canvas bitmap pixel = 1 CSS px) or assumes that the developer has made their canvas hidpi (which can be approximated with devicePixelRatio or done precisely using ResizeObserver).

I think it would be nifty if these definitions could somehow be worked into the existing definition of "natural dimensions", but that's beyond my spec-fu.

I'm not an expert in the CSS/HTML sizing stuff either, but I think you'll want a different concept than "natural dimensions". Imagine for example <canvas layoutsubtree><img src="/api/v1/web-embed/proxy?proxyUrl=https%3A%2F%2Fgithub.com%2Fwhatwg%2Fhtml%2Fpull%2Fmy5x5.png&mode=full" style="width:10px"></canvas>. The img is an "object" with "natural dimensions" of 5x5, but it's also an HTML element with a layout size of 10x10 (CSS) px (IDK the spec term for this). Since HIC views the DOM subtree and not the object, I think it should see a size of 10x10 (or 10*devicePixelRatio or whatever)

The "natural size" makes no assumptions about the CSS-to-canvas-grid ratio.

Agreed that we can't just use "natural dimensions" outright, but this does feel like a related concept so I hope it can be proximal to "natural dimensions".

As I understand it, the key problem is that we can't say "border box width in CSS pixels" because that's not what it is. It's actually the border box width/height multiplied by the ratio of output bitmap pixel size to CSS pixel size. Is that actually sufficiently descriptive to use?

We have the necessary language down below in the "Update the Rendering" section. I think the cleanest way to do this would be to create concepts called "snapshot width" and "snapshot height", then define them using the language from "Update the Rendering", and then you can refer to the concept every time you need this particular width and height.

@schenney-chromium Snapshots already have a width and height, see right after the definition of "element image snapshot". See also the line starting with "Let width be element's border box's width in CSS pixels multiplied by scaleX".

Does this already do the right thing, then?

The argument should be ElementOrElementImage, and the error conditions need to be updated as well. Same goes for OffscreenCanvas.updateElementGeometry, below.

The argument already is (Element or ElementImage) in the IDL and the "get element image snapshot" algorithm handles both, so this already seems correct to me. Do I misunderstand?

I agree with @foolip.

Slightly wrong now, I think? Only drawable elements (and their descendants) are laid out, not all canvas descendants.

We should have a definition for the drawable element, so we're not having to refer to them verbosely in several places. The explainer just uses "drawable element", which I think works, and then also "drawable subtree" for the descendants that aren't part of another drawable tree.

I suspect we want to reuse the same language as "capture the image" in View Transitions here, with appropriate adjustments (only rendering the drawable subtree, probably adjusting which properties are ignored). It even has the right pattern for the nested drawables.

Also, probably want to remove the "border box" text here. The border-box is used to define the origin of the drawing coords, and the default width/height, but the snapshot isn't actually limited to it in any way.

I suspect we want to reuse the same language as "capture the image" in View Transitions here, with appropriate adjustments (only rendering the drawable subtree, probably adjusting which properties are ignored). It even has the right pattern for the nested drawables.

I don't agree with this. The view transitions capture is different in almost all of the details that it discusses. captureElementImage cannot capture the document root, does not necessarily capture ink overflow, still captures scrollbars, does not apply effects like opacity and filters from the ancestor chain that would otherwise affect the rendering. And we do not produce an image.

Also, probably want to remove the "border box" text here. The border-box is used to define the origin of the drawing coords, and the default width/height, but the snapshot isn't actually limited to it in any way.

We do want to remove border box but it needs to be replaced with some clear statement like "everything within the src rect that is generated by drawing the element and its descendants".

Source rect has no place here, there is no source rect associated with paint snaphsot. The paint snapshot has no clipping, and it always includes all sources of ink overflow regardless of extent. For example, if the element has a box-shadow with blur, then the paint snapshot includes all the information necessary to rasterize the element including the infinite box shadow.

Isn't this just the element argument? This feels like a strange bit of indirection. (We will need to fetch the originating element if an ElementImage is passed, tho.)

See my comment above.

Why are we capturing a snapshot here, when we're just fetching data from the element?

The "get element image snapshot" algorithm only gets the snapshot from the Element-or-ElementImage argument, it doesn't capture a snapshot. That only happens in the rendering loop before "paint" events are fired.

I think Tab's point is that if step 1 is get element image snapshot given element then the snapshot's element must be element, then we just get the element again and set something on it. Why go round the circle? Is it to handle Element or ElementImage, in which case the algorithm should switch on which one we have.

If <var>element</var> is an Element:
  Set originatingElement to <var>element</var>
Otherwise
  Set originatingElement to <var>element</var>'s <span data-x="concept-element-image-snapshot-originating-element">originating element</span>

It is roundabout and targetElement can end up being the same as element, but the snapshot is also how we make sure that the element belongs to the canvas (this). I've tried to do all checks like this, and never check if there's a drawable attribute or walk a tree to check preconditions in the API, instead leaving that entirely to snapshot creation time in the rendering loop.

Does that match the Chromium implementation? Concretely, can elm.updateElementGeometry() be called immediately after elm is inserted as a canvas descendent, or does the rendering loop have to spin once first?

Assuming sourceRect is passed to the snapshot-capturing algo instead, the mention of it here can be dropped; the image you get will be perfectly sized already.

Also, "specified rectangular area" isn't actually specified. You need to construct this area from dx/y/w/h.

Yeah, you need to define the destination rect and then refer to it here.

There's similar language (search "specified rect") for clearRect() and fillRect(), but it's not as obvious here, so I'll use more words.

Does this imply that the top left corner of the source isn't necessarily painted at (dx, dy), but might be dx+dw and/or dy+dh if w/h are negative? Or is it always at dx/dy and you ignore the sign of width/height, or something else?

It means the top left may not be at dx/dy, yes. This is a deliberate choice to match the behavior of drawImage.

Yeah, should be elements, not "HTML elements", so SVG and MathML can be drawable.

Only drawable descendants.

We lay out everything under a content=drawable canvas because technically we need to. Other browsers would also have this requirement.

only if {preserveElementGeometry:true} wasn't passed

Also, it's updated from updateElementGeometry() even if not drawn.

Yeah. Maybe use "has non-null canvas transform" or "is in the drawn elements list" or whatever that concept was.

That doesn't seem right, "transformation matrix" is a concept, not the name of a property.

Move this up to right below width and height to keep the attributes together.