-
Notifications
You must be signed in to change notification settings - Fork 437
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[YouTube] Refactor player clients, add support for poTokens, extract visitor data from the service and more #1272
base: dev
Are you sure you want to change the base?
[YouTube] Refactor player clients, add support for poTokens, extract visitor data from the service and more #1272
Conversation
Also update client version and device info and add TVHTML5 client and WEB_EMBEDDED_PLAYER constants, these will be used in the future. TVHTML5_SIMPLY_EMBED_CLIENT_VERSION has been kept in its place as the corresponding client does not work anonymously anymore, so it will be removed soon.
This allows their internal usage in an upcoming new class to be used on other InnerTube clients than the WEB one.
This internal class, YoutubeStreamHelper, has to be public in order to be used by subpackages of the service's root one. It supports poTokens, HTML5 signatureTimestamp property, embed context and multiple InnerTube clients. It is meant to replace the corresponding methods in YoutubeParsingHelper, in order to reduce the class' size, code duplicates and improve its readability. It adds a new way to get age-restricted videos' streams, the only ones which are playable in YouTube embeds, which is not very common.
Also improve detection of age-restricted statuses in playability error messages returned by the service and provide version 7 of DASH manifests.
- Use POST requests with the same body as official clients do; - Update methods checking the client streaming URLs come from: - Replace TVHTML5_SIMPLY_EMBEDDED_PLAYER by TVHTML5; - Add WEB_EMBEDDED_PLAYER.
This new class, InnertubeClientRequestInfo, composed of two mutable subclasses, ClientInfo and DeviceInfo, allows to store client and device info in a better way, without requiring to pass more than 10 method parameters like in YoutubeStreamHelper currently. Mutability has been added in order to allow changing some fields easily, especially visitorData.
As YouTube is disabling ability to use a random visitor ID in a visitorData on player requests and BotGuard challenges, it shouldn't matter if we use a random one or not for other request types.
Some clients like TVHTML5 are not allowed on the visitor_id endpoint (with this client, a 400 HTTP response is returned with a precondition check failed error). Also disable pretty printing for these requests, like we do for others.
…some fixes visitorData are get using InnertubeClientRequestInfo and YoutubeParsingHelper.prepareJsonBuilder, which is replacing the corresponding method in YoutubeStreamHelper, removed in this commit. Also fix some bugs like JsonBuilder usages in some places and remove HLS manifest filtering for the iOS client, as we still use for now the full player response, in the case having streams requiring poTokens after some time as a last resort is useful.
This argument, which has been forgot, is required to get valid streaming URLs with this client.
This commits provides methods to get InnertubeClientRequestInfo instances, which can be used by extractor clients to get visitor data to pass to PoTokenProvider implementations using YoutubeParsingHelper. Ability to create custom instances has been removed, but returned objects can be modified. This is what YoutubeStreamHelper now uses to set the visitorData property.
Also make YoutubeParsingHelper.getOriginReferrerHeaders public, in order to be used by other extractor classes and improve the name of a parameter of YoutubeParsingHelper.getVisitorDataFromInnertube.
…creators Also use TVHTML5 user agent for requests from this client in these DASH manifests creators.
private static PoTokenProvider poTokenProvider; | ||
private static boolean fetchIosClient; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm sorry for a sort of a drive-by comment, but is it possible to not make these static
? For example, have them in YoutubeService
and pass them along when getting the stream extractor? It's trivial to make them static downstream if needed, and in the typical usage, where the YoutubeService
is already static, they will also be effectively static with something like ServiceList.YouTube.setPoTokenProvider(...)
.
My use case is that running the stream extractor is an expensive operation taking multiple seconds, so if I need to run multiple at the same time, I do it from multiple threads using different service and extractor instances.
It would be nice to not have to worry about synchronization here, reduce global state, and have the ability to have extractors with different providers.
This PR, which supersedes #1247, adds support for poTokens, refactor clients used and extract visitor data from the service itself, as generated visitor data do not work anymore to get valid player responses (rolling out in progress as of writing the first version of this PR description).
It also replaces age-restricted videos streams' fetching with the web embed player, which allows to play a very few age-restricted videos anonymously. This is the best we can do with what we can use (unless some method is still present and we haven't discovered it, but I have a lot of doubts).
TODO:
Fixes TeamNewPipe/NewPipe#11980.
Fixes TeamNewPipe/NewPipe#11803.
Fixes TeamNewPipe/NewPipe#11486.
Fixes partially TeamNewPipe/NewPipe#11382.