Automatic snapshot generator

[plehegar]

Idea here is to have the ability to generate snapshots based on git commits.

[xfq]

Currently, we can already link to snapshots with RawGit/HTMLPreview (without the WHATWG warning, tho), because they support using a specific tag or commit hash in the URL.

However, one issue with RawGit/HTMLPreview is, if the spec uses Bikeshed and auto-deploys the generated index.html to gh-pages with Travis CI, you need to find the corresponding commit in gh-pages, rather than using the commit in master directly.

---

Our friends at WHATWG do this in build-time: https://github.com/whatwg/whatwg.org/blob/e8cda18f1f9168b525f9612ce4ca49d7a92f6430/resources.whatwg.org/build/deploy.sh#L93-L103

They also include a client-side script in the boilerplate, letting the spec reader go to the commit snapshot by pressing y (similar to GitHub). (Bikeshed PR: speced/bikeshed#288)

[plehegar]

Here is what I think we'd need: provide a link to the commit snapshot, provide a history of snapshots, works with respec and bikeshed, easily deployable.

[xfq]

If we don't want to use RawGit/HTMLPreview, should we save the commit snapshots on GitHub (not in the default branch)? If so, we will have 1,000 separate files if there are 1,000 commits (and that doesn't even count the multiple-file spec case). PR Preview stores the specs on AWS, but AWS is not easy to access in China...

[xfq]

I just created a repo for this: https://github.com/xfq/commit-snapshot-tool

[plehegar]

if we need a cloud server, maybe we can use a labs.w3.org?

[tabatkins]

Note that pr-preview already exists and is almost this functionality; it works with Bikeshed and ReSpec and generates previews for any PR.

[xfq]

Note that pr-preview already exists and is almost this functionality; it works with Bikeshed and ReSpec and generates previews for any PR.

Yep. It uses a JSON file to reflect the configuration of the spec, and uses the Bikeshed CGI and Spec Generator with raw links on GitHub and RawGit to generate the spec, and saves the generated spec on AWS.

[xfq]

And drafts.csswg.org also has this functionality, e.g., https://drafts.csswg.org/history/css-values-4/

(Source)

[xfq]

Maybe we can reuse some code in the projects above, and deploy it to labs.w3.org.

[xfq]

@tobie Could you provide some hint on how we should build this tool (or even better, we can collaborate on extending pr-preview)?

I think we should monitor the push event, detect the config of the repo (either by reusing .pr-preview.json, or by creating a new config file), build the spec using Bikeshed/ReSpec's web service, and save the generated snapshot on some server. We probably also need frontend code to display a history of snapshots of a given spec.

[tobie]

I agree you can probably reuse large chunks of pr-preview for this. And the overall architecture you describe looks good to me.

I'd be happy to help, but that would require setting up some consultancy agreement which I doubt W3C has funds for.

PR Preview stores the specs on AWS, but AWS is not easy to access in China...

Oh! That's interesting. Is the filtering happening on the domain name or on the actual IP? If it's the former, adding custom domain names is easy. That's what we've done for WHATWG, e.g.: https://whatpr.org/html/3714/iframe-embed-object.html is hosted on AWS S3.

[xfq]

PR Preview stores the specs on AWS, but AWS is not easy to access in China...

Oh! That's interesting. Is the filtering happening on the domain name or on the actual IP? If it's the former, adding custom domain names is easy. That's what we've done for WHATWG, e.g.: https://whatpr.org/html/3714/iframe-embed-object.html is hosted on AWS S3.

It's a sophisticated system, consisting of DNS filtering, IP blocking, TCP ports blocking, and other blocking methods. Some (but not all) AWS servers are not accessible in some regions/ISPs in China, and the list of servers being blocked is constantly changing.

For example, in my ISP:

• The DNS seems to correctly resolve whatpr.org and pr-preview.s3.amazonaws.com
• I can successfully ping whatpr.org and pr-preview.s3.amazonaws.com (ICMP works correctly)
• I can't establish TCP connections to whatpr.org now (so that HTTP and SSH connections are blocked), but I can establish TCP connections to pr-preview.s3.amazonaws.com