startingOffset issues:

[kashyapsodha]

I tried executing the project locally using Spark Shell but facing issues with data displays in the terminal. In the default startingOffset setting which is 'latest', the output is a blank table but if I change the startingOffset setting to 'earliest', the output table does display 20-row output.
Please help me out.

[rhagarty]

@ScrapCodes Can you please take a look?

[ScrapCodes]

Did you push records after the kafka streaming job is started in spark shell, and waited for the processing duration?

[kashyapsodha]

Are you referring to the tail command?
I executed the spark-shell inside the command line generated by the 'tail' command.

[ScrapCodes]

Yes, tail command. You must execute it in another terminal, after the spark shell job has been submitted and you are waiting for the output.

[kashyapsodha]

Executed as instructed but 'Batch: 0" displays nothing. Post-execution of tail command on another terminal got output.
Will the table update every time I execute the tail command or should it update automatically after the first tail command?
Also, how do I validate if the output is correct?

[ScrapCodes]

Thanks for letting us know, that you got output, after inserting the data using another terminal.

Table updates automatically.

In order to validate the output, try to manually insert each row using kafka console send, please see the documentation on kafka(https://kafka.apache.org/0102/documentation.html#quickstart_send).