Pyblish DAG

[tokejepsen]

Goal

Provide a framework working with Pyblish plugins as nodes in a DAG.

Motivation

The current system of ordering plugins for predictable sequential execution has served Pyblish well, but it has got its drawbacks:

• Overview.
  • It can be very tricky to get an overview of the publishing pipeline when you have plugins relying on other plugins, and what data flows through the plugins.
• Adding new plugins.
  • The first question when adding a new plugin into a complex pipeline is always: "Where?".
• Chain plugins.
  • A lot of what Pyblish plugin do, are to compartmentalize code to execute on repeating data sets. With the current ordering system, it can be tricky to know when to execute a plugin if the data set relies on other plugins execution.

Sharing Plugins

I see these drawbacks as also hindering the Pyblish plugin ecosystem from being shareable. Taking a plugin from someone else's pipeline and inserting directly into your own, are destined for modification. How do you make sure that the external plugin operates on the desired data?
With ordering you will need to make sure that the order of the plugin fits into your pipeline, and already then you are modifying the code.
With a DAG system you could just insert the plugin into the pipeline and connect to your existing plugins without any code modification.

Multithread/multiprocess

By chaining plugins we have a predictable set of plugins that can be separated and put into another process or thread, to speed up the publishing.

Imlementation

The recent years there has been an increase in node based setups in DCCs where Nuke and Houdini are worth mentioning. From these systems we can make a list of what is required from a DAG implementation of Pyblish:

• Deferred Evaluation
  • Only evaluate when asked to.
• Reusability
  • The same plugin should be able to be used multiple times in multiple places in the graph.
• Graphs need to shareable.
  • Nuke does this very well, where you can copy and send graph as plain text, to be paste into a different session of Nuke.

Existing solutions

Unfortunately I have not encountered a framework for Python that allows for the flexibility we need for this to work. Explorations into adopting existing solutions have been made. These explorations were motivated by #143, which generated this issue.

For the lack of existing solutions I started experimenting and arrived at a proof of concept.
I believe this framework can be developed into its own identity, and maybe fill the visual programming gap for Python fueled by Pyblish DAG.
The mentality of the framework being that "Everything is a method, that return data".

For UI there are existing solution we can take advantage of, and worth mentioning are; https://github.com/LeGoffLoic/Nodz and https://github.com/mfessenden/SceneGraph.

[mottosso]

Some food for thought.

• http://trevorius.com/scrapbook/uncategorized/python-dependency-graph/
• http://forums.pyblish.com/t/pyblish-dag/105

[tokejepsen]

An interesting library; https://github.com/spotify/pythonflow

[mottosso]

Yes, it was all looking well, until I noticed they have ZeroMQ as a dependency.. good luck distributing anything for Maya using that.. Unfortunate! Almost curious about how tied it is exactly, and how much work it would involve to strip that dependency. It's clearly there for performance reasons (millions to trillions of tasks; they are Spotify) whereas we'd only need if for tens to hundreds.

[tokejepsen]

We use ZeroMQ to distribute the data processing

This seems to me that it's only for distribution of the processing that they depend on ZeroMQ. Which could indicate that we could use other distribution methods for processing.
There is no mention of ZeroMQ in the install requirements, so I think the library is pure python.

When I get some time I'll dig deeper into this library and try with a prototype for Pyblish.

[mottosso]

Ah, that would make sense.

I was thrown off by this.

[tokejepsen]

Hmmm, you may be right about that the library is dependent on ZeroMQ, but the code doesn't seem to need it. Will see what happens when I take for a test drive.

[davidlatwe]

Add one repo that is DAG related
PaulSchweizer/flowpipe

[mottosso]

Mentioning this here as they are somewhat related.

• Visualise Relationships of Assets getavalon/core#159

[PaulSchweizer]

Hi guys,

I created an example on how to convert a pyblish context into a flowpipe graph, it's actually pretty straight forward, please take a look at this gist:

https://gist.github.com/PaulSchweizer/0256b942406a9e765b2a3858b09cf8c4

This is just a conversion, meaning that the pyblish context is analyzed and then assembled into an according flowpipe graph with a node for each plugin and instance. It does not need any implementation in pyblish itself! We are currently mainly using this technique to push pyblish to our render farm (see below) but this also addresses most of the initial points raised in this ticket.

In general flowpipe offers a very simple api that allows for arbitrary node graph creation.

Things to note:

1. The collection step is not part of the graph itself as the flowpipe graph is per se a "static" graph meaning it does not have any functionality for scheduling nodes on the fly. Collection is done beforehand, the resulting instances are then mapped out as the graph.

2. Erroring Validations: Since the flowpipe graph is built before validation, it contains all nodes that WOULD be executed if all validations passed. The way I am currently addressing failing validations is to subsequently just bypass those instances that have failed. Another approach would be to do the validation beforehand as well.

3. Flowpipe requires data passed between nodes to be JSON-serializable. This decision was taken to ensure that the graph can be serialized easily so it can be sent to a remote destination for execution (e.g. render farm). This is why the example arbove makes sue to serialize the pyblish data in between nodes, even though that is of course overkill when evaluating the graph locally. There would be an easy way to loosen the restrictions in flowpipe to allow for any objects to be passed, but this can be looked into once the need arises. In general, a serialization of the context and instances would be great to have built into pyblish (unless I am overlooking it).

4. Since the graph is fully serializable and predictable, submission to a render farm is just another conversion step. I have successfully implemented that conversion for two different render farms so far and prepared an example on how this can be achieved for any render farm that supports dependencies, please take a look here: vfx_render_farm_conversion.py

5. Concurrency: We're currently implementing optional threading-based concurrency for local graph evaluation. So as long as the nodes in the graph themselves don't violate any threading principles like accessing shared objects and such, local concurrency would come for free. If that would work with pyblish out of the box is not guaranteed though, I haven't looked into it yet.

6. I am surely not taking all possible cases that pyblish offers into account, for example I have ignored custom ordering within a step, but I'm sure these things can also be achieved within the flowpipe graph. I plan to look deeper into those more edgy cases but am happy to get feedback on what you guys think might not work with this approach.

7. I prepared a few more examples in regards to flowpipe for VFX that might be of general interest, please take a look here:

• flowpipe for vfx pipelines
• flowpipe examples

[mottosso]

That's slick :) Thanks for sharing