Design a scalable approach to "viewer/blueprint commands"

[emilk]

We have a desire to trigger certain actions from our SDK that affects the viewer directly:

• Activate blueprint $ID
• Open/close recording $ID
• Play/pause/move time cursor
• …

So far, only a few of these have been implemented, and in rather ad-hoc ways:

• https://github.com/rerun-io/rerun/pull/8809/files

• rerun/crates/store/re_protos/proto/rerun/v0/log_msg.proto

  Lines 16 to 17 in 026d846

  	// A message that contains a blueprint activation command.
  	BlueprintActivationCommand blueprint_activation_command = 3;

Where can we send these commands?

• From the logging SDKs (Python, Rust, C++)
• From Python notebooks (via logging SDK)
• From JS API ?

Blueprint changes

Some of these commands can be encoded as changes to the blueprint store.
This include setting the time state, once this is part of the blueprint.

Changes to blueprint can be implemented as any other store changes, i.e. by pushing a chunk (recordbatch).
Similarly, querying the blueprint state ("what is the time cursor at?") can be implemented as a data query.

Design

None yet, but it would be nice with something data driven, e.g. a protobuf definition of all commands as an oneof.

Related issues

• Allow configuring selected timeline & initial time from blueprint #7728
• Direct frame / sequence jump URL parameters (and CLA parameters) #8318

[Wumpf]

Any change we formulate as a change in blueprint will - if not handled in a special way - be an action that is on the undo stack in the viewer. Either we should come up with a mechanism that adds exceptions to that or treat some things to be outside of the blueprint.

[jleibs]

I believe it's important here to find a good way to logically decouple the concepts of:

• Streaming Data
• Configuring Blueprints
• Controlling the Viewer

Our current implementation of Blueprint lifecycle very much straddles the Data / Viewer divide in a way that makes this confusing to reason about.

There are good motivations for how we got where we are, driven by the desire to declare the way the viewer should be configured from the same context where data is being produced. Strongly coupling that with the stream of data itself. In particular, this makes workflows like .save() work quite nicely.

I suspect some part in the answer here makes sense when thinking about a single-producer multi-consumer setup.

• Assume you have a single robot streaming data, and two different employees looking at that data in their respective viewer instances.
  • One one hand, it's nice if the robot is able to declare things like default views and styling, and so sending a blueprint associated with that stream which both users receive makes sense.
  • But it's rare (until we start thinking about collaboration use-cases) that the source should be orchestrating which recording is open, or over-riding the time-point that's selected. These seem more control patterns that, even if controlled programmatically are tied to a single analysis/exploration task.

My proposal is that the initial "viewer control" SDK should be decoupled from the recording stream and only apply to contexts where the SDK was used to spawn the viewer.

A pattern like:

viewer = rr.spawn()

viewer.ctrl(...)

makes this somewhat explicit, and avoids situations where a viewer losers control of its interaction paradigm because it connected to an arbitrary stream just to see the data.

[Wumpf]

I very sympathetic to the idea of "viewer control data" which is strictly client specific and believe we can go a step further and combine it with application settings. Application settings as a concept are fairly hidden right now and are the once piece that is already fully application specific. In a nutshell advantages of that would be:

• no new source of data
  • we already have a system in place to handle "local settings" and are just reusing this
  • we already have a concept for when and how this is persisted (-> open the same recording on the same machine, you end up at the same time & timeline! 👍 )
• viewer control is naturally able to control other intuitively local-only aspects like "is this viewer maximixed" or zoom level
• full instrumentation of a viewer instances/installations is highly desirable since it allows users to automate setups

Only drawback I can think of right now (there gotta be more) is the application settings screen becomes a partial view into this. But that's fine, it already is! E.g. zoom is persisted in the exact same way and doesn't show up there.

Furthermore, I believe we can still have overlapping data in the blueprint. Time control settings could well reside in either. It does however open case-by-case questions of who overrides whom, might not always be clear (intuitively application settings/viewer control are the most highlevel though).