-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
b5251a6
commit 4538a2e
Showing
2 changed files
with
13 additions
and
113 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,122 +1,24 @@ | ||
# Provenance as a Service | ||
# Test Suite and SQAaaS | ||
|
||
yProv is a provenance service aimed at addressing multi-level provenance as well as reproducibility challenges in climate analytics experiments. It allows scientists to manage provenance information compliant with the [W3C PROV standard](https://www.w3.org/TR/prov-overview/) in a more structured way and navigate and explore the provenance space across multiple dimensions, thus enabling the possibility to get coarse- or fine-grained information according to the level of interest. | ||
|
||
yProv is a joint project between [University of Trento](https://www.unitn.it) and [CMCC](https://www.cmcc.it). | ||
|
||
The deployment consists of two Docker containers: | ||
- **yProv** Web Service front-end | ||
- **Neo4J** graph database engine back-end | ||
|
||
### Preliminary setup | ||
|
||
- Create a named volume to make Neo4j data persistent | ||
``` | ||
docker volume create neo4j_data | ||
``` | ||
- Create a named volume to export logs to the host machine | ||
``` | ||
docker volume create neo4j_logs | ||
``` | ||
- Create a named volume to make yProv configuration and data persistent to the host machine | ||
The unit and mock tests are split into two files, auth and documents, respectively, to respect the division of the swagger documentation files. | ||
To run the tests you need to start Docker Desktop locally and run the command in any folder of the project: | ||
``` | ||
docker volume create yprov_data | ||
pytest | ||
``` | ||
Create a Docker network to enable communication between the two Docker containers | ||
- Create a Docker network | ||
or if you want a more detailed output: | ||
``` | ||
docker network create yprov_net | ||
pytest -v | ||
``` | ||
|
||
### Deployment from DockerHub | ||
### SQAaaS | ||
|
||
- Run the Neo4j container | ||
``` | ||
docker run \ | ||
--name db \ | ||
--network=yprov_net \ | ||
-p 7474:7474 -p 7687:7687 \ | ||
-d \ | ||
-v neo4j_data:/data \ | ||
-v neo4j_logs:/logs \ | ||
-v $HOME/neo4j/import:/var/lib/neo4j/import \ | ||
-v $HOME/neo4j/plugins:/plugins \ | ||
--env NEO4J_AUTH=neo4j/password \ | ||
--env NEO4J_ACCEPT_LICENSE_AGREEMENT=eval \ | ||
-e NEO4J_apoc_export_file_enabled=true \ | ||
-e NEO4J_apoc_import_file_enabled=true \ | ||
-e NEO4J_apoc_import_file_use__neo4j__config=true \ | ||
-e NEO4J_PLUGINS=\[\"apoc\"\] \ | ||
neo4j:enterprise | ||
``` | ||
- Run the yProv container | ||
``` | ||
docker run \ | ||
--restart on-failure \ | ||
--name web \ | ||
--network=yprov_net \ | ||
-p 3000:3000 \ | ||
-d \ | ||
-v yprov_data:/app/conf \ | ||
--env USER=neo4j \ | ||
--env PASSWORD=password \ | ||
hpci/yprov:latest | ||
``` | ||
Every time a push is made to the SQA branch, the Github Action QAA file is invoked. It automatically starts the SQAaaS platform pipeline and runs mock tests inside it. Unit tests are instead run locally, so it is always necessary to start docker desktop. The file automatically generates the volumes, containers and docker network and removes them once the files have finished running. The file uses the hpci-yprov:mocktest image to run mock tests inside the SQAaaS platform, the dockerfile code is in the hpci-yprov:mocktest folder. | ||
|
||
### Deployment from sources | ||
|
||
You can also build the yProv image by your own (using the available ```Dockerfile```) and start the two containers. | ||
|
||
- Build the yProv image | ||
``` | ||
docker build . -t yprov_web | ||
``` | ||
- Run the bash script to start the two containers | ||
``` | ||
./deploy_service.sh | ||
``` | ||
|
||
## yProv CLI | ||
yProv includes a `Command Line Interface` to interact with the service in a more convenient way. | ||
Please refer to [this documentation](https://github.com/HPCI-Lab/yProv-CLI/tree/main) to install the CLI. | ||
|
||
## Get started | ||
You can find some ready-to-use examples to get started with yProv under the `examples` folder. | ||
|
||
## Advanced options | ||
|
||
#### Load Neo4j plugins | ||
###### Method 1: ```NEO4J_PLUGINS``` utility | ||
You can configure the Neo4j deployment to automatically download and install plugins. The ```NEO4J_PLUGINS``` environment variable can be used to specify the plugins to install. | ||
|
||
*Example: Install the APOC Core plugin (```apoc```) and the Graph Data Science plugin (```graph-data-science```)* | ||
|
||
You can use the Docker argument | ||
``` | ||
--env NEO4J_PLUGINS='["apoc", "graph-data-science"]' | ||
``` | ||
|
||
###### Method 2: Installing plugins | ||
To install plugins, including user-defined procedures, mount the folder or volume containing the plugin JARs to ```/plugins```, for example: | ||
``` | ||
docker run \ | ||
... | ||
-v $HOME/neo4j/plugins:/plugins \ | ||
neo4j:enterprise | ||
``` | ||
Neo4j automatically loads any plugins found in the ```/plugins``` folder on startup. | ||
### Future works | ||
|
||
#### Change secret key | ||
A secret key is used to encode/decode the JSON Web Token for user authentication. The default secret key can be overridden by setting the ```SECRET_KEY``` environment variable: | ||
The SQAaaS platform is not yet able to handle multi-container architectures like yProv, however it is working on a solution that can include a docker-compose file. This approach has already been implemented and the code is contained in the docker-compose and dockerfile files in the root folder. To run it use the commands: | ||
``` | ||
docker run \ | ||
--name web \ | ||
--network=yprov_net \ | ||
-p 3000:3000 \ | ||
-d \ | ||
-v yprov_data:/app/conf \ | ||
--env USER=neo4j \ | ||
--env PASSWORD=password \ | ||
--env SECRET_KEY=my_secret_key \ | ||
hpci/yprov:latest | ||
docker-compose build --no-cache | ||
docker-compose up | ||
``` | ||
There is also an image that allows you to run both unit tests and mock tests inside a single container and this is contained in the hpci-yprov:1.3 folder. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters