Provenance as a Service

yProv is a provenance service aimed at addressing multi-level provenance as well as reproducibility challenges in climate analytics experiments. It allows scientists to manage provenance information compliant with the W3C PROV standard in a more structured way and navigate and explore the provenance space across multiple dimensions, thus enabling the possibility to get coarse- or fine-grained information according to the level of interest.

yProv is a joint project between University of Trento and CMCC.

The deployment consists of two Docker containers:

• yProv Web Service front-end
• Neo4J graph database engine back-end

Preliminary setup

• Create a named volume to make Neo4j data persistent

docker volume create neo4j_data

• Create a named volume to export logs to the host machine

docker volume create neo4j_logs

• Create a named volume to make yProv configuration and data persistent to the host machine

docker volume create yprov_data

Create a Docker network to enable communication between the two Docker containers

• Create a Docker network

docker network create yprov_net

Deployment from DockerHub

• Run the Neo4j container

    docker run \
        --name db \
        --network=yprov_net \
        -p 7474:7474 -p 7687:7687 \
        -d \
        -v neo4j_data:/data \
        -v neo4j_logs:/logs \
        -v $HOME/neo4j/import:/var/lib/neo4j/import \
        -v $HOME/neo4j/plugins:/plugins \
        --env NEO4J_AUTH=neo4j/password \
        --env NEO4J_ACCEPT_LICENSE_AGREEMENT=eval \
        -e NEO4J_apoc_export_file_enabled=true \
        -e NEO4J_apoc_import_file_enabled=true \
        -e NEO4J_apoc_import_file_use__neo4j__config=true \
        -e NEO4J_PLUGINS=\[\"apoc\"\] \
        neo4j:enterprise

• Run the yProv container

    docker run \
        --restart on-failure \
        --name web \
        --network=yprov_net \
        -p 3000:3000 \
        -d \
        -v yprov_data:/app/conf \
        --env USER=neo4j \
        --env PASSWORD=password \
        hpci/yprov:latest

Deployment from sources

You can also build the yProv image by your own (using the available Dockerfile) and start the two containers.

• Build the yProv image

    docker build . -t yprov_web

• Run the bash script to start the two containers

   ./deploy_service.sh 

yProv CLI

yProv includes a Command Line Interface to interact with the service in a more convenient way. Please refer to this documentation to install the CLI.

Get started

You can find some ready-to-use examples to get started with yProv under the examples folder.

Advanced options

Load Neo4j plugins

Method 1: NEO4J_PLUGINS utility

You can configure the Neo4j deployment to automatically download and install plugins. The NEO4J_PLUGINS environment variable can be used to specify the plugins to install.

Example: Install the APOC Core plugin (apoc) and the Graph Data Science plugin (graph-data-science)

You can use the Docker argument

--env NEO4J_PLUGINS='["apoc", "graph-data-science"]'

Method 2: Installing plugins

To install plugins, including user-defined procedures, mount the folder or volume containing the plugin JARs to /plugins, for example:

docker run \
   ...
   -v $HOME/neo4j/plugins:/plugins \
   neo4j:enterprise

Neo4j automatically loads any plugins found in the /plugins folder on startup.

Change secret key

A secret key is used to encode/decode the JSON Web Token for user authentication. The default secret key can be overridden by setting the SECRET_KEY environment variable:

    docker run \
        --name web \
        --network=yprov_net \
        -p 3000:3000 \
        -d \
        -v yprov_data:/app/conf \
        --env USER=neo4j \
        --env PASSWORD=password \
        --env SECRET_KEY=my_secret_key \
        hpci/yprov:latest