Having problems using javafx with complex pipeline

[gb91757]

Hi,
Not sure if this is a java issue or gstreamer but I have the following working command on (Windows 10):

gst-launch-1.0 -e autovideosrc num-buffers=1000 ! videoconvert ! tee name=t ! queue ! videorate ! video/x-raw,framerate=20/1 ! x264enc pass=4 quantizer=21 ! queue ! mpegtsmux ! queue ! filesink location=test.ts t. ! queue ! autovideosink

I've modified the FXCamera.java example to use this pipeline:

Bin bin2 = Gst.parseBinFromDescription( "autovideosrc num-buffers=1000 ! videoconvert ! tee name=t ! queue ! videorate ! video/x-raw,framerate=20/1 ! x264enc pass=4 quantizer=21 ! queue ! mpegtsmux ! queue ! filesink name=dst t. ! queue", true);

(The file location is added later in the code using a timestamp)

Then added the autovideosink programatically:
Element sink = ElementFactory.make("autovideosink", "sink"); pipeline.addMany(bin2, sink); System.out.println("linked: " + Pipeline.linkMany(bin2, sink));

This works as I would expect - blank javafx image and a new window opened by gstreamer showing the live webcam and a file recorded until window is closed.

Now I try to use the FXImageSink:
Element sink = imageSink.getSinkElement(); pipeline.addMany(bin2, sink); System.out.println("linked: " + Pipeline.linkMany(bin2, sink));

Now GStreamer fails:

ERROR 1 Internal data stream error. BaseSrc: [autovideosrc1-actual-src-ksvide]
Bus Message : GstMessageError, gerror=(GError)NULL, debug=(string)"../libs/gst/base/gstbasesrc.c(3072):\ gst_base_src_loop\ ():\ /GstPipeline:pipeline0/GstBin:bin1/GstAutoVideoSrc:autovideosrc1/GstKsVideoSrc:autovideosrc1-actual-src-ksvide:\012streaming\ stopped,\ reason\ not-negotiated\ (-4)", details=(structure)"details,\ flow-return=(int)-4;";
Bus Message : GstMessageStreamStatus, type=(GstStreamStatusType)GST_STREAM_STATUS_TYPE_ENTER, owner=(GstElement)"(GstQueue)\ queue4", object=(GstTask)"(GstTask)\ queue4:src";
Bus Message : GstMessageStreamStatus, type=(GstStreamStatusType)GST_STREAM_STATUS_TYPE_ENTER, owner=(GstElement)"(GstQueue)\ queue3", object=(GstTask)"(GstTask)\ queue3:src";
ERROR 10 Could not create handler for stream Element: [mpegtsmux0]
Bus Message : GstMessageError, gerror=(GError)NULL, debug=(string)"../gst/mpegtsmux/mpegtsmux.c(1011):\ mpegtsmux_create_streams\ ():\ /GstPipeline:pipeline0/GstBin:bin1/MpegTsMux:mpegtsmux0";
Bus Message : GstMessageStreamStatus, type=(GstStreamStatusType)GST_STREAM_STATUS_TYPE_ENTER, owner=(GstElement)"(GstQueue)\ queue5", object=(GstTask)"(GstTask)\ queue5:src";
Bus Message : GstMessageStateChanged, old-state=(GstState)GST_STATE_READY, new-state=(GstState)GST_STATE_PAUSED, pending-state=(GstState)GST_STATE_VOID_PENDING;

Strangely if I edit the bin and remove the three elements:

x264enc pass=4 quantizer=21 ! queue ! mpegtsmux !

The FX image appears correctly (my file is rubbish of course as I've simply removed the encoding and muxing)

I'm finding this totally confusing and just wondering if there is anything I can do with the java source to add some debug to find out why? Not sure why the appsink is getting affected by the stream going to the file as it's split off by the tee. But it appears that the appsink maybe mis-behaving in this more complicated scenario (or at least gstreamer thinks it is!).

Thanks for any pointers that might help me debug this.

Andrew.

[neilcsmith-net]

The first thing I would try is using Gst.parseLaunch(..) to create the full pipeline, including appsink name=sink. The get the appsink using getElementByName and pass it in to the FXImageSink constructor. Can get some weird effects with complex pipelines in a Bin.

The other thing is adding a videoconvert in each path after the tee. Looking at gst-inspect-1.0 x264enc it doesn't look like there's an overlap in the formats supported, which there could be with the auto sink.

[gb91757]

Hi Neil,

That was quick!

And apologies for using this discussion rather than the google group - I was rushing to get something out before the weekend.
I wanted to put the appsink in the bin as the command had it in the middle originally but stupily I couldn't see a way to "update" an appsink - but now from your comment above I realise how easy it is and I feel really stupid!

[neilcsmith-net]

No problem, and you're welcome! 😄 I enabled the discussions here so it could be used anyway - might make it the default in future. Google Groups has its own issues.

[gb91757]

I made the changes you suggest. No errors now but it just hangs with a blank screen. Start again on Monday I think.

[gb91757]

Hi Neil,
Coding (like driving) when tired (and frustrated) is not a good idea! I'd somehow commented out a very important line at my last attempt. Your suggestions above fixed my problem first time I think. For completeness here is the final working code:

`

    Bin bin = Gst.parseBinFromDescription(
         "autovideosrc num-buffers=1000 ! tee name=t ! queue ! videoconvert ! videorate ! video/x-raw,framerate=20/1 ! " +
          "x264enc pass=4 quantizer=21 ! queue ! mpegtsmux ! queue ! filesink name=dst t. ! queue ! videoconvert ! appsink name=sink",
          true);
    
    /**
     * FXImageSink from gst1-java-fx wraps the native data from the
     * GStreamer AppSink in a JavaFX image. Requested dimensions will be set
     * on the caps for the AppSink.
     */
    FXImageSink imageSink = new FXImageSink((AppSink)bin.getElementByName("sink"));
    imageSink.requestFrameSize(640, 480);

    pipeline.add(bin);
    `

Thanks for your great help as usual.

Andrew.

[neilcsmith-net]

Excellent! And thanks for letting us know - always good to know when something works out.