Task based docker

.gitignore

@@ -19,6 +19,9 @@ coverage
 # Compiled binary addons (http://nodejs.org/api/addons.html)
 build/Release
 
+# Bikeshed clone
+bikeshed
+
 # Dependency directory
 # Commenting this out is preferred by some people, see
 # https://www.npmjs.org/doc/misc/npm-faq.html#should-i-check-my-node_modules-folder-into-git-

.npmrc

@@ -0,0 +1 @@
+prefix = ${HOME}/node_modules

DOCKER.md

@@ -0,0 +1,135 @@
+# Run Publican in Docker
+
+If you have [Docker installed](https://docs.docker.com/installation/) and running, you can run it in a few commands.
+
+It can summarize as;
+
+```bash
+docker pull webspecs/publican:latest
+docker run -it --rm -p 7002:7002 -v "$(pwd)/data":/srv/webplatform/specs/data -v "$(pwd)/spec-data":/opt/bikeshed/bikeshed/spec-data webspecs/publican:latest bin/run.sh
+```
+
+If you want to run with your own customizations, you’d have to do also have [**Docker Compose** installed](https://docs.docker.com/compose/install/) and do the following.
+
+Follow the instructions in **Run using _Docker Compose_** in [the registry description instructions](https://registry.hub.docker.com/u/webspecs/publican/)
+
+
+## Other possible usage
+
+### Build Docker container
+
+When we call `docker pull webspecs/publican:latest`, its either calling a last successful Docker build command that has the `-t webspecs/publican`
+or its asking to get the latest version from the Docker hub.
+
+In order to deploy, we need to have an image to run, that’s why you have to build one.
+
+To build a **webspecs/publican** container, you need to use the publican repository.
+
+Once a build is successful you can push it to [Push Docker container to Docker Hub registry](#Push Docker container to Docker Hub registry).
+
+Once you have a clone of publican, get the dependencies;
+
+    make clone-bikeshed
+
+Which will clone bikeshed and create required directories for you.
+
+Build the container
+
+    make docker-build
+
+Run the container
+
+    make docker-run
+
+If you review what’s in the [Makefile](./Makefile), you’ll see that it basicall expands to something like this;
+
+    docker run -it --rm -p 7002:7002 \
+           -v /Users/renoirb/workspaces/webplatform/service-publican/repo/data:/srv/webapps/publican/data \
+           -v /Users/renoirb/workspaces/webplatform/service-publican/repo/spec-data:/opt/bikeshed/bikeshed/spec-data \
+           webspecs/publican:latest bin/run.sh
+
+Notice that the `-v /full/path/host:/full/path/container` option requires FULL path in both sides of the column
+
+The last part of the `docker run` is what’s going to run, in this case `bin/run.sh`.
+Which means you could get into a running container shell if you need to.
+
+That’s what the `make docker-bash` do.
+
+Refer to the appropriate documentation if you want to [install](https://docs.docker.com/installation/#installation) and run this project from a Docker container.
+
+
+### Run an instance locally
+
+Run a container, and send a hook call
+
+        make docker-run
+
+On another terminal tab, send a hook;
+
+        curl -H "Content-Type: application/json" -XPOST localhost:7002/hook -d '{"base_ref":"master","repository":{"name":"assets","owner":{"name":"webspecs"}}}'
+
+
+you should see something like;
+
+        {"ok":true,"details":"Queued 1431380611165-28770219 for processing."}
+
+And in the container, you’d see;
+
+![publican-run-hook](https://cloud.githubusercontent.com/assets/296940/7575805/f18cb1d2-f805-11e4-8ec3-dba68dae7785.png)
+
+
+
+### Enter the container shell
+
+You can initialize an empty workspace using `publican.js init`, it’ll write in `data/` that should already be mounted through the `Makefile`.
+
+Its also useful to understand what’s happening behind the scenes.
+
+        make docker-bash
+
+![publican-init](https://cloud.githubusercontent.com/assets/296940/7575777/b5a76176-f805-11e4-99e7-3a7c58dd304a.png)
+
+
+
+### Push Docker container to Docker Hub registry
+
+The Docker registry is basically a storage service of docker container states from which we can pull and run from it.
+It saves us to rebuild the container on every server that would run the container.
+
+Docker uses a semantic similar to git, a container can be commited and pushed to a repository. That’s what we’ll do here.
+
+Refer to [Docker command line "commit" documentation](http://docs.docker.com/reference/commandline/cli/#commit-a-container) for further details.
+
+This project Docker registry entry is at [registry.hub.docker.com/u/**webspecs/publican**](https://registry.hub.docker.com/u/webspecs/publican/).
+
+To commit a container, we need a running one. Once you have one, you can commit.
+
+Get current running container; that’s what we commit.
+
+```bash
+docker ps
+
+CONTAINER ID        IMAGE                             COMMAND                CREATED             STATUS              PORTS                    NAMES
+3cce48d3b30c        servicepublican_publican:latest   "/bin/bash /srv/webp   11 minutes ago      Up 11 minutes       0.0.0.0:8002->7002/tcp   sandboxpublican_web_1"
+```
+
+If you have write access as a *collaborator* to the *webspecs* organization within Docker hub; Commit and push
+
+```bash
+docker commit 3cce48d3b30c
+docker push webspecs/publican
+```
+
+Done.
+
+
+
+### Run from Docker composer
+
+TODO, see notes in [**webspecs/publican**](https://registry.hub.docker.com/u/webspecs/publican/) Docker hub description.
+
+
+## Reference
+
+Refer to the appropriate documentation if you want to [install](https://docs.docker.com/installation/#installation)
+and run this project from a Docker container locally.

Dockerfile

@@ -0,0 +1,67 @@
+#
+# Publican Docker runner
+#
+# See also:
+#   * https://github.com/nodesource/docker-node/blob/master/ubuntu/trusty/node/0.10.36/Dockerfile
+
+FROM nodesource/trusty:0.10.36
+
+MAINTAINER Renoir Boulanger <[email protected]>
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+# Dependencies: Bikeshed, PhantomJS, Bikshed’s lxml
+RUN apt-get update && apt-get -y upgrade && \
+    apt-get install -yqq git python2.7 python-dev python-pip libxslt1-dev libxml2-dev zlib1g-dev && \
+    apt-get install -yqq libfontconfig1 libfreetype6 curl && \
+    apt-get autoremove -yqq --purge && \
+    pip install --upgrade lxml
+
+# Copy everything we have locally into the container
+# REMINDER: Make sure you run `make clone-bikeshed`, we prefer to keep a copy locally outside
+# of the data volume. Otherwise it would make problems saying that bikeshed clone is not in the
+# same filesystem.
+COPY . /srv/webapps/publican/
+
+# Make sure we have a "non root" user and
+# delete any local workbench data/ directory
+RUN /usr/sbin/groupadd --system --gid 990 webapps && \
+    /usr/sbin/useradd --system --gid 990 --uid 990 -G sudo --home-dir /srv/webapps --shell /bin/bash webapps && \
+    sed -i '/^%sudo/d' /etc/sudoers && \
+    echo '%sudo ALL=NOPASSWD: ALL' >> /etc/sudoers && \
+    mv /srv/webapps/publican/bikeshed /opt && \
+    rm -rf data && \
+    mkdir -p data/temp && \
+    rm -rf Dockerfile Makefile .git .gitignore DOCKER.md && \
+    chown -R webapps:webapps /srv/webapps/publican && \
+    chown -R webapps:webapps /opt/bikeshed
+
+# Switch from root to webapps system user
+# It **HAS to be** the SAME uid/gid as the owner on the host from which we’ll use as volume
+USER webapps
+
+# Where the session will start from
+WORKDIR /srv/webapps/publican
+
+# Environment variables
+ENV PATH /srv/webapps/publican/node_modules/.bin:/srv/webapps/publican/bin:/srv/webapps/publican/.local/bin:$PATH
+ENV HOME /srv/webapps/publican
+ENV TMPDIR /srv/webapps/publican/data/temp
+ENV NODE_ENV production
+ENV GIT_DISCOVERY_ACROSS_FILESYSTEM true
+
+# Run what `make deps` would do
+RUN pip install --upgrade --user --editable /opt/bikeshed && \
+    mkdir -p node_modules && npm install
+
+# Declare which port we expect to expose
+EXPOSE 7002
+
+# Allow cli entry for debug, but make sure docker-compose.yml uses "command: bin/run.sh"
+ENTRYPOINT ["/bin/bash"]
+
+# Note leftover: Ideally, it should exclusively run
+#ENTRYPOINT ["/bin/bash", "/srv/webapps/publican/bin/run.sh"]
+
+# Note leftover: What it ends up doing
+#CMD ["node_modules/forever/bin/forever", "--fifo", "logs", "0"]

Makefile

@@ -0,0 +1,25 @@
+SHELL := bash
+
+default: deps
+
+clone-bikeshed: datadirs
+		git clone -b webspecs https://github.com/webspecs/bikeshed.git bikeshed
+
+datadirs:
+		mkdir -p $(CURDIR)/data/{queue,gits,logs,temp}
+		mkdir -p $(CURDIR)/spec-data
+
+# Make sure Dockerfile has equivalent at note about `make deps`
+deps:
+		pip install --user --editable bikeshed
+		npm install
+
+docker-build:
+		docker build -t webspecs/publican .
+
+docker-bash:
+		docker run -it --rm -v "$(CURDIR)/data":/srv/webapps/publican/data -v "$(CURDIR)/spec-data":/opt/bikeshed/bikeshed/spec-data -p 7002:7002 webspecs/publican:latest
+
+docker-run:
+		docker run -it --rm -v "$(CURDIR)/data":/srv/webapps/publican/data  -v "$(CURDIR)/spec-data":/opt/bikeshed/bikeshed/spec-data -p 7002:7002 webspecs/publican:latest bin/run.sh
+

README.md

@@ -1,4 +1,4 @@
-publican
-========
+# Publican
 
 The set of tools used to automate publishing in the WebSpecs project
+

bin/run.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+
+export RUNDIR="/srv/webapps/publican"
+
+cd $RUNDIR
+
+node_modules/forever/bin/forever start $RUNDIR/bin/server.js
+node_modules/forever/bin/forever --fifo logs 0

data/config.json.dist

@@ -0,0 +1,18 @@
+{
+    "bikeshed":     "/opt/bikeshed/bikeshed.py"
+,   "rsyncPath":    "/srv/webapps/publican/"
+,   "python":       "python2"
+,   "logFile":      "logs/all.log"
+,   "email":        {
+        "to":       "[email protected]"
+    ,   "from":     "[email protected]"
+    ,   "host":     "localhost"
+    ,   "password": "XXXX not the real thing, will not work. Should be replaced by a login that can email from here."
+    ,   "ssl":      true
+    ,   "tls":      false
+    ,   "level":    "error"
+    ,   "handleExceptions": true
+    }
+,   "purgeAllURL":  "https://api.fastly.com/service/not_the_real_thing/purge_all"
+,   "purgeAllKey":  "not_the_real_thing"
+}

docker-compose.yml.sample

@@ -0,0 +1,29 @@
+main:
+  image: webspecs/publican:latest
+
+  # Default in this container is bash, using command below runs
+  # publican server for us while allowing us to enter the original
+  # container image in case of need to understand what’s available.
+  command: bin/run.sh
+
+  # Make sure you have a data folder in the same
+  # directory as this file. Container expects the following
+  # files and folders.
+  #  - data/{queue,gits,logs,temp}
+  #  - data/config.json
+  #  - spec-data/read-only
+  volumes:
+    - data/:/srv/webapps/publican/data
+    - spec-data/:/opt/bikeshed/bikeshed/spec-data
+
+  restart: always
+
+  ports:
+    - "7002:7002"
+
+  dns:
+    - 10.10.10.41
+    - 8.8.8.8
+
+# vim: et ts=2 sw=2 ft=yaml:
+

lib/lock.js

@@ -3,19 +3,19 @@ var lock = require("lock")()
 ,   timeout = 1000 * 60 * 10 // 10 minutes
 ;
 
-exports.createLock = function (man) {
-    return function (key, action, done) {
+exports.createLock = function createLockHandler (man) {
+    return function createLockReturner (key, action, done) {
         man.log.info("Locking " + key);
-        lock(key, function (release) {
+        lock(key, function createLockLocker (release) {
             man.log.log("silly", "Processing locked key " + key);
             var tid = setTimeout(function () {
                 man.log.error("Timeout on lock expired, releasing " + key + " with risk of conflict.");
                 release(done)();
             }, timeout);
-            action(function (err) {
+            action(function createLockLockerAction (err) {
                 man.log.info("Releasing lock " + key);
                 clearTimeout(tid);
-                release(function () { done(err); })();
+                release(function createLockRelease () { done(err); })();
             });
         });
     };