The database is no up-to-date

[calixteman]

For Firefox nightly, the last available build is 20181025100246.
According to https://archive.mozilla.org/pub/firefox/nightly/2018/10/ we should have builds:

• 20181025220044
• 20181026100128
• 20181026220839
  Because of that, some of my tools are broken.

[peterbe]

It seems, that there's just a big delay in Buildhub. Really big.
Today is Monday and now they're there. For example https://mozilla-services.github.io/buildhub/?q=20181025220044&products[0]=firefox

As I was working on a fix to PollBot last Friday I discovered the same thing:

Also, I did discover that some of the PollBot stuff fails horribly for Beta. In particular for version 64.0b4 which, still at the time of writing, hasn't made it to Buildhub even though I can see the files in archive.mozilla.org. Sigh. I'm going to let the cron jobs (that pick up the slack when Lambda jobs fail) go another 24 hours.

That seems to have "worked". I.e. being super patient.

[peterbe]

So there is Buildhub2 which solves almost all of these problems. Granted, (legacy) Buildhub should work in real-time but it doesn't (...very well). Probably because it's fragile and unreliable. Buildhub2 solves this by taking a much simpler approach. All new versions are shipped with a file called buildhub.json (or suffixed with that at least, like firefox-66-buildhub.json) and that project uses a SQS queue that "makes sure" that failure to process doesn't happen. Granted, bugs can still happen or super unfortunate luck in saving to the database but Buildhub2 has a brand new cron job pattern that does a much better, and much faster, job of fixing what might have fail in the run-time pipe.

The code is complete, the Stage server is ready, and buildhub.json files are published for all en-US builds. The only thing blocking us is the buildhub.json files for l10n repacks. That work is blocked on this:
https://bugzilla.mozilla.org/show_bug.cgi?id=1459302

[peterbe]

PS, I know you know most of this ^ @calixteman but I thought it was a good summary for others to read if they're curious or have forgotten the status of this stuff.

[calixteman]

Before friday, I think buildhub was almost real-time.
I began to write a scraper to workaround the issue (in waiting for buildhub2) but I didn't manage to find where are the buildids for beta and release.
So any chance to have a fix or need I to continue to write a scraper (I don't feel like writing it) ?

[peterbe]

To be honest, it's extremely unlikely that we'll fix this. I suspect that one of the Lambda jobs failed. They do sometimes. And we ended up at the mercy of the cron job which not only runs only every 24h but also takes 3-4 hours to complete.

For some positive change, try sharing your work case on that bugzilla bug to inspire it being worked on.

[willkg]

If I discover Buildhub is missing build xyz and I find the relevant files on archive.mozilla.org, can someone get buildhub to index that build? Is there any tooling for that or do we just hope it finds it eventually?

[peterbe]

If I discover Buildhub is missing build xyz and I find the relevant files on archive.mozilla.org, can someone get buildhub to index that build? Is there any tooling for that or do we just hope it finds it eventually?

Yes. You can either ping someone in CloudOps or file a bug. There's a cron job that runs every 6 hours. Whoever wants to do this, just needs to copy exactly what that 6h cron job does and run it right away.

[wlach]

I was just investigating this in the context of some weird mission control numbers we were seeing on nightly mac/linux. It looks like we don't have data for that platform/channel combination since November 5th. Consequently the dashboard is saying (correctly) that nightly info is current not available:

I've filed a bug to look into this: https://bugzilla.mozilla.org/show_bug.cgi?id=1506862

[pascalchevrel]

https://missioncontrol.telemetry.mozilla.org/#/release/windows show 63.0.1 as the last column while we shipped 63.0.3 yesterday, is that issue the cause of this bug?

[wlach]

I believe so. It appears buildhub is missing quite a few windows builds on release as well, now that I look at it:

https://mozilla-services.github.io/buildhub/?platform[0]=win32

My guess is that it's the same issue.

[peterbe]

I could well be. We've discovered another unrelated bug in Buildhub1 that might lead to missed builds. Oh my! We're actively working on it. A patch is ready. We just need to do some final testing.

The current challenge is that we need to be smart regarding legacy builds that makes a firefox-0.0a0.L10N.win0.installer.exe AND firefox-0.0a0.L10N.win0.exe.