Skip to content

Commit

Permalink
Readme file
Browse files Browse the repository at this point in the history
  • Loading branch information
mattiaBarbo committed Sep 12, 2024
1 parent b5251a6 commit 4538a2e
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 113 deletions.
124 changes: 13 additions & 111 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,122 +1,24 @@
# Provenance as a Service
# Test Suite and SQAaaS

yProv is a provenance service aimed at addressing multi-level provenance as well as reproducibility challenges in climate analytics experiments. It allows scientists to manage provenance information compliant with the [W3C PROV standard](https://www.w3.org/TR/prov-overview/) in a more structured way and navigate and explore the provenance space across multiple dimensions, thus enabling the possibility to get coarse- or fine-grained information according to the level of interest.

yProv is a joint project between [University of Trento](https://www.unitn.it) and [CMCC](https://www.cmcc.it).

The deployment consists of two Docker containers:
- **yProv** Web Service front-end
- **Neo4J** graph database engine back-end

### Preliminary setup

- Create a named volume to make Neo4j data persistent
```
docker volume create neo4j_data
```
- Create a named volume to export logs to the host machine
```
docker volume create neo4j_logs
```
- Create a named volume to make yProv configuration and data persistent to the host machine
The unit and mock tests are split into two files, auth and documents, respectively, to respect the division of the swagger documentation files.
To run the tests you need to start Docker Desktop locally and run the command in any folder of the project:
```
docker volume create yprov_data
pytest
```
Create a Docker network to enable communication between the two Docker containers
- Create a Docker network
or if you want a more detailed output:
```
docker network create yprov_net
pytest -v
```

### Deployment from DockerHub
### SQAaaS

- Run the Neo4j container
```
docker run \
--name db \
--network=yprov_net \
-p 7474:7474 -p 7687:7687 \
-d \
-v neo4j_data:/data \
-v neo4j_logs:/logs \
-v $HOME/neo4j/import:/var/lib/neo4j/import \
-v $HOME/neo4j/plugins:/plugins \
--env NEO4J_AUTH=neo4j/password \
--env NEO4J_ACCEPT_LICENSE_AGREEMENT=eval \
-e NEO4J_apoc_export_file_enabled=true \
-e NEO4J_apoc_import_file_enabled=true \
-e NEO4J_apoc_import_file_use__neo4j__config=true \
-e NEO4J_PLUGINS=\[\"apoc\"\] \
neo4j:enterprise
```
- Run the yProv container
```
docker run \
--restart on-failure \
--name web \
--network=yprov_net \
-p 3000:3000 \
-d \
-v yprov_data:/app/conf \
--env USER=neo4j \
--env PASSWORD=password \
hpci/yprov:latest
```
Every time a push is made to the SQA branch, the Github Action QAA file is invoked. It automatically starts the SQAaaS platform pipeline and runs mock tests inside it. Unit tests are instead run locally, so it is always necessary to start docker desktop. The file automatically generates the volumes, containers and docker network and removes them once the files have finished running. The file uses the hpci-yprov:mocktest image to run mock tests inside the SQAaaS platform, the dockerfile code is in the hpci-yprov:mocktest folder.

### Deployment from sources

You can also build the yProv image by your own (using the available ```Dockerfile```) and start the two containers.

- Build the yProv image
```
docker build . -t yprov_web
```
- Run the bash script to start the two containers
```
./deploy_service.sh
```

## yProv CLI
yProv includes a `Command Line Interface` to interact with the service in a more convenient way.
Please refer to [this documentation](https://github.com/HPCI-Lab/yProv-CLI/tree/main) to install the CLI.

## Get started
You can find some ready-to-use examples to get started with yProv under the `examples` folder.

## Advanced options

#### Load Neo4j plugins
###### Method 1: ```NEO4J_PLUGINS``` utility
You can configure the Neo4j deployment to automatically download and install plugins. The ```NEO4J_PLUGINS``` environment variable can be used to specify the plugins to install.

*Example: Install the APOC Core plugin (```apoc```) and the Graph Data Science plugin (```graph-data-science```)*

You can use the Docker argument
```
--env NEO4J_PLUGINS='["apoc", "graph-data-science"]'
```

###### Method 2: Installing plugins
To install plugins, including user-defined procedures, mount the folder or volume containing the plugin JARs to ```/plugins```, for example:
```
docker run \
...
-v $HOME/neo4j/plugins:/plugins \
neo4j:enterprise
```
Neo4j automatically loads any plugins found in the ```/plugins``` folder on startup.
### Future works

#### Change secret key
A secret key is used to encode/decode the JSON Web Token for user authentication. The default secret key can be overridden by setting the ```SECRET_KEY``` environment variable:
The SQAaaS platform is not yet able to handle multi-container architectures like yProv, however it is working on a solution that can include a docker-compose file. This approach has already been implemented and the code is contained in the docker-compose and dockerfile files in the root folder. To run it use the commands:
```
docker run \
--name web \
--network=yprov_net \
-p 3000:3000 \
-d \
-v yprov_data:/app/conf \
--env USER=neo4j \
--env PASSWORD=password \
--env SECRET_KEY=my_secret_key \
hpci/yprov:latest
docker-compose build --no-cache
docker-compose up
```
There is also an image that allows you to run both unit tests and mock tests inside a single container and this is contained in the hpci-yprov:1.3 folder.
2 changes: 0 additions & 2 deletions tests/test_auth.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
def test_auth_register():

# correct new username and password
"""
payload = {
"user": "userA",
"password": "userA"
Expand All @@ -27,7 +26,6 @@ def test_auth_register():
}
response = requests.post(PATH + '/register', json=payload)
assert response.status_code == 201
"""

# username and password already exist
payload = {
Expand Down

0 comments on commit 4538a2e

Please sign in to comment.