Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[YouTube] Refactor player clients, add support for poTokens, extract visitor data from the service and more #1272

Open
wants to merge 17 commits into
base: dev
Choose a base branch
from

Conversation

AudricV
Copy link
Member

@AudricV AudricV commented Feb 1, 2025

This PR, which supersedes #1247, adds support for poTokens, refactor clients used and extract visitor data from the service itself, as generated visitor data do not work anymore to get valid player responses (rolling out in progress as of writing the first version of this PR description).

It also replaces age-restricted videos streams' fetching with the web embed player, which allows to play a very few age-restricted videos anonymously. This is the best we can do with what we can use (unless some method is still present and we haven't discovered it, but I have a lot of doubts).

TODO:

  • describe in a more detailed way changes in this PR?
  • add Javadocs
  • update mocks

Fixes TeamNewPipe/NewPipe#11980.
Fixes TeamNewPipe/NewPipe#11803.
Fixes TeamNewPipe/NewPipe#11486.
Fixes partially TeamNewPipe/NewPipe#11382.

Also update client version and device info and add TVHTML5 client and
WEB_EMBEDDED_PLAYER constants, these will be used in the future.

TVHTML5_SIMPLY_EMBED_CLIENT_VERSION has been kept in its place as
the corresponding client does not work anonymously anymore, so it will
be removed soon.
This allows their internal usage in an upcoming new class to be used on
other InnerTube clients than the WEB one.
This internal class, YoutubeStreamHelper, has to be public in order to
be used by subpackages of the service's root one.

It supports poTokens, HTML5 signatureTimestamp property, embed context
and multiple InnerTube clients. It is meant to replace the
corresponding methods in YoutubeParsingHelper, in order to reduce the
class' size, code duplicates and improve its readability.

It adds a new way to get age-restricted videos' streams, the only ones
which are playable in YouTube embeds, which is not very common.
Also improve detection of age-restricted statuses in playability error
messages returned by the service and provide version 7 of DASH
manifests.
- Use POST requests with the same body as official clients do;
- Update methods checking the client streaming URLs come from:
  - Replace TVHTML5_SIMPLY_EMBEDDED_PLAYER by TVHTML5;
  - Add WEB_EMBEDDED_PLAYER.
This new class, InnertubeClientRequestInfo, composed of two mutable
subclasses, ClientInfo and DeviceInfo, allows to store client and
device info in a better way, without requiring to pass more than 10
method parameters like in YoutubeStreamHelper currently.

Mutability has been added in order to allow changing some fields
easily, especially visitorData.
As YouTube is disabling ability to use a random visitor ID in a
visitorData on player requests and BotGuard challenges, it
shouldn't matter if we use a random one or not for other
request types.
Some clients like TVHTML5 are not allowed on the visitor_id endpoint
(with this client, a 400 HTTP response is returned with a precondition
check failed error).

Also disable pretty printing for these requests, like we do for others.
…some fixes

visitorData are get using InnertubeClientRequestInfo and
YoutubeParsingHelper.prepareJsonBuilder, which is replacing the
corresponding method in YoutubeStreamHelper, removed in this commit.

Also fix some bugs like JsonBuilder usages in some places and remove
HLS manifest filtering for the iOS client, as we still use for now
the full player response, in the case having streams requiring poTokens
after some time as a last resort is useful.
This argument, which has been forgot, is required to get valid
streaming URLs with this client.
This commits provides methods to get InnertubeClientRequestInfo
instances, which can be used by extractor clients to get visitor data
to pass to PoTokenProvider implementations using YoutubeParsingHelper.

Ability to create custom instances has been removed, but returned
objects can be modified. This is what YoutubeStreamHelper now uses to
set the visitorData property.
Also make YoutubeParsingHelper.getOriginReferrerHeaders public, in
order to be used by other extractor classes and improve the name of a
parameter of YoutubeParsingHelper.getVisitorDataFromInnertube.
…creators

Also use TVHTML5 user agent for requests from this client in these
DASH manifests creators.
Comment on lines +105 to +106
private static PoTokenProvider poTokenProvider;
private static boolean fetchIosClient;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry for a sort of a drive-by comment, but is it possible to not make these static? For example, have them in YoutubeService and pass them along when getting the stream extractor? It's trivial to make them static downstream if needed, and in the typical usage, where the YoutubeService is already static, they will also be effectively static with something like ServiceList.YouTube.setPoTokenProvider(...).

My use case is that running the stream extractor is an expensive operation taking multiple seconds, so if I need to run multiple at the same time, I do it from multiple threads using different service and extractor instances.
It would be nice to not have to worry about synchronization here, reduce global state, and have the ability to have extractors with different providers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ASAP Issue needs to be fixed as soon as possible bug Issue is related to a bug enhancement New feature or request youtube service, https://www.youtube.com/
Projects
None yet
2 participants