Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sporadic update PRs with invalid checksums for SourceForge URLs #168

Closed
Zlopez opened this issue Apr 21, 2021 · 13 comments · Fixed by #219
Closed

Sporadic update PRs with invalid checksums for SourceForge URLs #168

Zlopez opened this issue Apr 21, 2021 · 13 comments · Fixed by #219
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@Zlopez
Copy link

Zlopez commented Apr 21, 2021

Hi,

today the flathubbot opened a strange PR on org.gnome.Chess. PR is opened for dependency which doesn't have x-data-checker options set up and it just changed the hash without changing the URL. I looked at the URL and the file had last update in 2016.

I will close the PR for now, because the build is failing anyway.

@gasinvein

This comment has been minimized.

@gasinvein
Copy link
Collaborator

gasinvein commented Apr 21, 2021

There are multiple other apps hit by this at the same time, and seems like all affected sources are hosted on SourceForge.
flathub/org.octave.Octave#142
flathub/org.gnome.Evince#47
flathub/org.gnome.Chess#28
flathub/org.gnome.Books#26
flathub/org.freedesktop.Sdk.Extension.texlive#45
flathub/org.flightgear.FlightGear#39
flathub/net.gwyddion.Gwyddion#23
flathub/me.kozec.syncthingtk#34
flathub/io.github.peazip.PeaZip#5
flathub/im.riot.Riot#175
flathub/com.sweethome3d.Sweethome3d#22
flathub/com.chez.GrafX2#5
And the invalid new checksum is the same one everywhere: 251a85b3bac687974f360d3796048c20ded3bf0bd69e0d1cfd1db23d013f89ed

@Vanadiae
Copy link

Thanks @gasinvein for mentioning the affected MRs here :)

@gasinvein
Copy link
Collaborator

Any idea what has happened? Probably some SourceForge issue? Looking at flathub download cache, it seems like this has occurred before.

@SISheogorath
Copy link

Source forge seems to have no permalinks to source code and/or employing some sort of blocking which results in the external data checker getting confused.

@gasinvein
Copy link
Collaborator

Given that download-sources job with the incorrect checksum has completed successfully for Element, libsoup/libbcurl didn't detect any http error either. Which means that SourceForge did actually sent invalid response without any errors (no incomplete read or anything). How can we catch this case? Is it possible at all?

@gasinvein gasinvein added the help wanted Extra attention is needed label Apr 21, 2021
@SISheogorath
Copy link

Iirc they send a valid page that is just HTML. At least that's what I got last time. So it's more of a figuring out how to use source forge than external a data checker issue.

@wjt
Copy link
Contributor

wjt commented Jun 16, 2021

This hit me for the first time today :) flathub/org.tuxpaint.Tuxpaint#15 – I wish we had the contents of the offending page, given that we could add a hack…

@wjt
Copy link
Contributor

wjt commented Jun 16, 2021

And the invalid new checksum is the same one everywhere: 251a85b3bac687974f360d3796048c20ded3bf0bd69e0d1cfd1db23d013f89ed

Oh, I missed this point before.

I know it's an awful kludge but maybe we should denylist this checksum.

Another idea:

https://github.com/flathub/flatpak-external-data-checker/blob/master/src/lib/utils.py#L116-L129

There is no check that the response code is 2xx. I wonder if the response code would turn out to be 5xx in this failure case.

@gasinvein
Copy link
Collaborator

gasinvein commented Jun 16, 2021

There is no check that the response code is 2xx. I wonder if the response code would turn out to be 5xx in this failure case.

aiohttp raises an exception by default when it receives status 4xx or 5xx.
We ask aiohttp to raise an exception on receiving response with error status here:
https://github.com/flathub/flatpak-external-data-checker/blob/435d2d5de3d72833d7300c0b0ccf2ff2bb637b64/src/lib/externaldata.py#L362-L363
I believe we have tests covering that.

@gasinvein
Copy link
Collaborator

I know it's an awful kludge but maybe we should denylist this checksum.

I'm considering this as the last resort. Maybe we could check Content-Type header? Clearly it shouldn't be text/html for an archive. Although would be good if we could check what's the Content-Type of the erroneous sourceforge response.

@gasinvein gasinvein changed the title Strange PR opened by flathubbot on org.gnome.Chess Sporadic update PRs with invalid checksums for SourceForge URLs Jul 23, 2021
@gasinvein gasinvein pinned this issue Jul 23, 2021
@gasinvein
Copy link
Collaborator

We've captured the erroneous page contents. As of time of whiting this post, it's available in flathub download cache. The contents are:

<html><head>
<title>SourceForge</title>
<!-- <script src="/js/jquery.com/jquery-1.11.0.min.js"></script> -->
<script src="https://code.jquery.com/jquery-1.11.0.min.js"></script>
<script src="https://sourceforge.net/js/mirrors.js"></script>
<script src="/js/sf.js"></script>
<script>
var DR_loc = DR_parse_hash_url();
if (DR_loc) {
    DR_sf_main(DR_loc);
} else {
    window.location.href = 'https://sourceforge.net/home.html';
}
</script>
<meta name="description" content="Free, secure and fast downloads from the largest Open Source applications and software directory - SourceForge.net"/>
</head><body>
<noscript>
We're sorry -- the Sourceforge site is currently in Disaster Recovery mode.  Please check back later.
</noscript>
</body></html>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants