All I have to say is thank you for doing this! This is truly super cool. It’s a rare kind of gem to see thoughtful work like this in our new AI era.

so a comment right above yours asks a question I'm curious about too, and I wish that person asked you, so I'm copying it here:

--- Does this also work when the search space becomes large? I fear this might lead to an exponential blowup.

(The examples shown are all tiny) ---

Someone else asked something similiar but basically, how does / should this work when candidate drop targets are many e.g. when dragging a first or second rectangle onto a page in Figma when no relationships have obviously been set.

Out of curiosity, does your library support physical springs for animation between states? That’d be a great touch.

Very cool demo! One idea I had in the back of my mind is extending grammar of graphics for interactive visualisations. What you did might be a good way how to approach this! Did you think about doing something like this btw.?

Andrew, thanks for answering questions here. edflsafoiewq's reading of the twisted trees above (take the barycentric coordinates of the cursor between three drop states, then interpolate the UI) is the core of Tom Ngo's Embedded Constraint Graphics (ECG), which I worked on with Tom and Golan Levin at Interval Research in the 1990s (US patent 5933150, expired 2016).

I described ECG on HN just after the patent expired in 2016, when G9.js came up, which jy14898 linked above. Golan drew the faces for his Mouther in the ECG editor, by dragging the eyes and mouth around:

https://news.ycombinator.com/item?id=12572696

In ECG the states are drawn examples at the vertices of simplices, glued along shared faces into a simplicial complex. The state is a point in one simplex, and its barycentric coordinates are the blend weights. You drag any point on the blended picture, the pseudoinverse of the Jacobian turns the drag into a change of weights, and at a boundary the state crosses into the neighboring simplex.

A few parts of that might plug into Dragology: continuous blends between states instead of snapping to the nearest one, several handles held at once (the multitouch case in your limitations section), pinned points, and a complex that says which states may blend, so the author supplies a few examples instead of enumerating every reachable state. Your speculative renders could be the examples: render each candidate state, read off the positions, and that list of numbers is an ECG example.

I just put up an MIT licensed reimplementation with live demos. It starts with a talking face after Golan's Mouther (1995), builds the engine up from two pictures, then points it at a 3D head you turn by the nose, ARKit blendshape coefficients, and window layouts. Beside each drawing it shows the weights, a map of the state space, and the shape of the matrix the solver inverts.

ECG: The Whole Story (live demos):

https://hyperties.org/read/ecg/the-whole-story/

Source, no runtime dependencies:

https://github.com/SimHacker/WillWrightShowForFood/tree/main...

My notes on the limitations in section 9 of your paper and earlier work on each:

https://github.com/SimHacker/WillWrightShowForFood/blob/main...