Extend API to support stereo streams

[dzhoshkun]

With #5 GIFT-Grab will support stereo video stream capture from BlackMagic devices. When extending the API, among others, the implications on encoding should be considered.

Encoding is currently done by IVideoTarget implementors that expose only an append method (in addition to the update method inherited from IObserver). However both methods take only a VideoFrame as a parameter. Would it be good for instance to add an optional VideoFrame parameter to these methods? This has the advantage that VideoFrame stays as it is (=dumb, simple).

Another option is to extend VideoFrame itself. Possible design patterns are pointed out here: #5 (comment) Some initial thoughts as well: #5 (comment), #5 (comment) This however does not sound like a good option any more, as it changes VideoFrame's and its clients' interfaces.

[dzhoshkun]

The discussion with @cbergeles and his team on 23 Oct 2018 has revealed that a video frame should be as tightly coupled to its stereo frame as possible.
This is because stereo frames are hardware-synchronised, and as such if for instance they get encoded into two different video files, the sync information will essentially be lost.

[dzhoshkun]

• check out whether this could be useful: https://www.boost.org/doc/libs/1_67_0/libs/gil/doc/index.html

[cbergeles]

I watched the video, or parts of it, and I can't recommend pursuing this. It seems a bit esoteric... I've never heard of the library before, and personally I would not use it over OpenCV (with which most people have experience!). Just my 2 cents :)

[dzhoshkun]

@cbergeles Thanks! The particular interesting bit of GIL in this case is the pixel iterator. This with the aim of providing NumPy-like "views" on e.g. a video frame's stereo frame. Tom and I were looking into this, considering whether it could help extend the API in a minimally invasive way.

[dzhoshkun]

Re: OpenCV

I agree it's a great toolkit. And GIFT-Grab currently uses it for a number of tasks. However in the particular case of GIFT-Grab it's an overkill to ask the user to install OpenCV to be able to use GIFT-Grab. Using OpenCV merely for capturing and encoding video streams is a bit like driving a Ferrari in the countryside :) So we are in the process of removing the dependency on OpenCV (see #9).

[tvercaut]

Re Boost GIL, I agree it is a bit esoteric and probably a bit of an overkill as well but as @dzhoshkun mentioned, some of its underpinning concepts could help us support stereo in an elegant and easy manner.

We could offer a slightly different interface in python and c++ to make use of the most adequate tool for this in each language. For example, if a stereo image is provided in memory as a left+right concatenated together, depending on the exact memory layout, we could create a numpy view on the underlying array to show only the left or the right image by using adequate (very long) stride/padding:
https://www.numpy.org/devdocs/reference/generated/numpy.lib.stride_tricks.as_strided.html

OpenCV also support the notion of steps but I haven't looked at this in detail:
https://docs.opencv.org/master/d3/d63/classcv_1_1Mat.html

[dzhoshkun]

Idea:

• an optional stereo_count parameter for the init_from_specs() method
• and its corresponding getter as well!
• possibly an analogue of the above for this VideoFrame constructor
• an optional stereo_index parameter for the data() method
• in this configuration the data_length() method will report the length of a single frame, or the length of a specific frame by virtue of a similar optional index parameter

[dzhoshkun]

• extend the video frame NumPy compatibility tests to accommodate new interface as well

[dzhoshkun]

• is 3fdbc03 necessary?

[dzhoshkun]

• why are frames so green? seems to be related DeckLink 4K Extreme 12G returns BGRA frames with low alpha values #30, looks like it might not be BGRA after all, maybe a bug ?