Jekyll/Liquid generation is dead-slow because of tutorial ToC

[dragos]

All the time is spent in several nested loops computing the table of contents for tutorials. Unfortunately Liquid is a pretty poor language so there aren't any obvious workarounds, but it's definitely worth investigating. Once I commented those loops times went down to around 1s.

Maybe a solution using categories or tags would reduce the number of iterations (350 pages, but only 33 are part of the Scala tour).

$ bundle exec jekyll serve --profile --incremental
Configuration file: /Users/dragos/workspace/git/scala.github.com/_config.yml
            Source: /Users/dragos/workspace/git/scala.github.com
       Destination: /Users/dragos/workspace/git/scala.github.com/_site
 Incremental build: enabled
      Generating... 
    Liquid Warning: Liquid syntax error (line 164): Unexpected character { in "{{{` and `}}" in overviews/scaladoc/for-library-authors.md

Filename                                                           | Count |     Bytes |    Time
-------------------------------------------------------------------+-------+-----------+--------
_layouts/tutorial.html                                             |   132 | 20395.38K | 202.110
_includes/tutorial-tour-list.txt                                   |   133 | 18740.42K | 197.912
_includes/tutorial-toc.txt                                         |   132 | 18618.52K | 196.613
_layouts/overview-large.html                                       |   164 | 24041.61K | 151.205
_includes/toc-large.txt                                            |   164 | 20639.72K | 144.796
tutorials/index.md                                                 |     1 |   184.98K |   1.759
_layouts/overview.html                                             |    36 |  1177.23K |   1.444
es/overviews/index.md                                              |     1 |    41.87K |   0.729
_includes/localized-overview-index.txt                             |     1 |    41.71K |   0.728
_includes/header.txt                                               |   369 |  1777.55K |   0.456

[jarrodu]

I am not sure if speed is really something that need to optimized for. It is just an annoyance.

What is your motivation for working on it?

[SethTisue]

@jarrodwb getting the site build down time from minutes to seconds seems like a pretty sweet improvement to me, if it can be done!

[jarrodu]

Okay you convinced me. ;)

[SethTisue]

at scala/scala-lang#493 (comment) Simon seemed to be saying this was fixed in his fork, so it might be worth looking there and trying to find what the relevant change was

[heathermiller]

Just to respond to @dragos's description; yes I think there's a way to get rid of all of the loops. Back when I built this whole thing, there were no categories or tags built into Jekyll (they were only available as plugins, which were incompatible with GitHub pages, so we couldn't ever use them before). I had commented at length about this problem elsewhere in this repo in an issue (will look for it); but basically the difference is that categories seem to be an actual array of things under the name of your category that you can revisit rather than having to traverse every page on the whole site again and again and again.

[SethTisue]

I had commented at length about this problem elsewhere in this repo in an issue

probably this comment and this followup

[SethTisue]

I dug around and found this old description of the problem (and a solution) from Simon:

if you traverse every document ever written on every page that could need it, things will be slow.
my compile time is around 0.9 seconds per page by just not doing that. e. g. we have languages: [en, es, de, ko] already in the frontmatter, so we don't need to traverse all the files looking for a translation for every translation. my approach is slightly more low-tec, but it works: If I say that I have a translation, it just links to it.

[charafau]

I was talking with Simon the other day. This is what he said:

https://github.com/soc/scala-lang/blob/gh-pages/documentation/tutorial/java/index.md
languages: [en, de, es, ko]

It is not perfect (as you need to add translations manually) but I don't think that translations are added that often. (check raw file)

[adamvoss]

Link is broken since that repository does not exist anymore. Here is a mirror:

https://github.com/dwijnand/scala-lang/blob/soc/documentation/tutorial/java/index.md

[SethTisue]

current mirror is https://github.com/soc-mirror/scala-lang/blob/gh-pages/documentation/tutorial/java/index.md

[heathermiller]

Another thing we can do: if there is some kind of Jekyll plugin that'll help us speed things up, we could consider building/serving docs.scala-lang.org on an EPFL machine. Both hosting on GitHub and hosting on EPFL machines have their drawbacks, but one relative benefit of running on EPFL infrastructure is our ability to use stuff like plugins. Just brainstorming.

[SethTisue]

yay!!! 🎉