Merge branch 'main' into labelbox_integration

octo-technology · Apr 16, 2024 · 5cb97ae · 5cb97ae
2 parents 38b0929 + 1e25fb1
commit 5cb97ae
Show file tree

Hide file tree

Showing 100 changed files with 1,066 additions and 1,081 deletions.
diff --git a/README.md b/README.md
@@ -1,14 +1,18 @@
-# VIO - Visual Inspection Orchestrator
+<div align="center">
+    <h1>VIO - Visual Inspection Orchestrator</h1>
 
 ![CI edge_orchestrator](https://github.com/octo-technology/VIO/actions/workflows/ci_edge_orchestrator.yml/badge.svg)
 ![CI edge_interface](https://github.com/octo-technology/VIO/actions/workflows/ci_edge_interface.yml/badge.svg)
 ![GitHub issues](https://img.shields.io/github/issues/octo-technology/VIO)
 
-Visual Inspection Orchestrator is a modular framework made to ease the deployment of VI usecases.
+🎥 Visual Inspection Orchestrator is a modular framework made to ease the deployment of VI usecases 🎥
 
 *Usecase example: Quality check of a product manufactured on an assembly line.*
+</div>
 
-VIO full documentation can be found [here](https://octo-technology.github.io/VIO/)
+<h1></h1>
+
+## 🏗️ Modular framework
 
 The VIO modules are split between:
 
@@ -24,31 +28,35 @@ The VIO modules are split between:
 - [The hub monitoring](docs/hub_monitoring.md)
 - [The hub deployment playbook](docs/hub_deployment.md)
 
-## Requirements
+**VIO full documentation can be found [here](https://octo-technology.github.io/VIO/)**
+
+## 🧱 Requirements
 
 - `docker` installed
 - `make` installed
 
-## Install the framework
+## 🚀 Getting started
+
+### Install the framework
 
 ```shell
 git clone [email protected]:octo-technology/VIO.git
 ```
 
-## Run the stack
+### Running the stack
 
 To launch the stack you can use the [Makefile](../Makefile) on the root of the repository which define the different
-target based on the [docker-compose.yml](../docker-compose.yml):
+target based on the [docker-compose.yml](../docker-compose.yml) as described below, or [run the modules locally]().
 
-### Start vio
+#### Start vio
 
 To start all edge services (orchestrator, model-serving, interface, db) with local hub monitoring (grafana):
 
 ```shell
 make vio-edge-up
 ```
 
-### Stop vio
+#### Stop vio
 
 To stop and delete all running services :
 
@@ -60,6 +68,8 @@ To check all services are up and running you can run the command `docker ps`, yo
 
 ![stack-up-with-docker](docs/images/stack-up-with-docker.png)
 
+### Accessing the services 
+
 Once all services are up and running you can access:
 
 - the swagger of the edge orchestrator API (OrchestratoAPI): [http://localhost:8000/docs](http://localhost:8000/docs)
@@ -72,16 +82,32 @@ launch the following actions:
 
 ![vio-architecture-stack](docs/images/edge_orchestrator-actions.png)
 
-## Releases
+## 🛰️ Technology features
+- 🏠 Hosting :
+  - ☁️ Hub : Cloud possibilities with [Azure](https://portal.azure.com/#home) and [GCP](https://cloud.google.com/)
+  - 🛸 Host : Using raspberries
+  - 🐳 Host : [Docker](https://www.docker.com/) or locally with anaconda
+- 👮 Fleet management :
+  - 📦 Fleet integration/deployment with [Ansible](https://docs.ansible.com/ansible/latest/index.html)
+  - 🕵️ Fleet supervision/observability with [Grafana](https://grafana.com/) & [Open-Telemetry](https://opentelemetry.io/docs/)
+- ⚡️Backend API with [FastAPI](https://fastapi.tiangolo.com/)
+- 📜 Frontend with [Vue.js](https://fr.vuejs.org/)
+- 🏭 Continuous Integration & Continuous Development : 
+  - ♟️ Github actions
+  - 📝️ Clean code with [Black](https://black.readthedocs.io/en/stable/index.html) & [Flake8](https://flake8.pycqa.org/en/latest/) 
+  - ✅ Tested with [Pytest](https://docs.pytest.org/en/8.0.x/)
+- 📈 [Grafana](https://grafana.com/) insight & dashboard
+
+## 🏭 Releases
 
  Build Type        | Status                                                                                                                                                                                                    | Artifacts                                                           
 -------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------
  **Docker images** | [![Status](https://github.com/octo-technology/VIO/actions/workflows/publication_vio_images.yml/badge.svg)](https://github.com/octo-technology/VIO/actions/workflows/publication_vio_images.yml/badge.svg) | [Github registry](https://github.com/orgs/octo-technology/packages) 
 
-## License
+## 📝 License
 
 VIO is licensed under [Apache 2.0 License](docs/LICENSE.md)
 
-## Contributing
+## 🙋 Contributing
 
 Learn more about how to get involved on [CONTRIBUTING.md](docs/CONTRIBUTING.md) guide
diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md
@@ -1,37 +1,46 @@
 # Contributing
 
 ## Contribution rules
+
 - The code must be exhaustively tested.
 - Python test package: `pytest` (with possibility to use unittest mocks)
 - Code style: PEP8
 - Programming language for code and comments: English
 
 ## Coding conventions
+
 - 120 character max per line
 - Use python 3.6 `fstring` instead of `format()` or `%s`
 - Directories, filenames, function and method names in `snake_case`
 - Class names in `UpperCamelCase`
 - Private and protected function and method names prefixed with `_`
 - Please implement private function and method under the corresponding public for more readability
 - Variables name in `snake_case`, constants `MAJ_SNAKE_CASE`
-- Static method should be used only when needed (for example a method is called without the instance of the 
-corresponding class).
+- Static method should be used only when needed (for example a method is called without the instance of the
+  corresponding class).
 - Don't let any dead code. Prefer to create a ticket in the backlog board. Package `vulture` can be used to check any
- existing dead code.
-- Use [pathlib](https://docs.python.org/3/library/pathlib.html#module-pathlib) instead of native Python [os.path](https://docs.python.org/3/library/os.path.html)
+  existing dead code.
+- Use [pathlib](https://docs.python.org/3/library/pathlib.html#module-pathlib) instead of native
+  Python [os.path](https://docs.python.org/3/library/os.path.html)
 
 ## Exception conventions
+
 - Create a custom exception in the module `exception.py` as follow:
+
 ```python
 class MyCustomException(Exception):
     pass
 ```
+
 - Add a custom message when it is raised:
+
 ```python
 if not something:
     raise MyCustomException('My custom message.')
 ```
+
 - Log the custom exception message when it is caught via the `exception` method of the logger as follow:
+
 ```python
 try:
     do_something()
@@ -41,10 +50,13 @@ except MyCustomException as e:
 ```
 
 ## Logging conventions
+
 - The logger should always be used at the class level and not the module level.
 - The logger should always be created through the _getChild_ method as a class attribute.
 - The created child logger should always be used in the class.
-- Be aware that all nodes classes that inherit from AbstractNode have already a logger that should be used in the class scope
+- Be aware that all nodes classes that inherit from AbstractNode have already a logger that should be used in the class
+  scope
+
 ```python
 from aivi.tools.logger import logger
 
@@ -54,14 +66,16 @@ class MyClassWithLogging:
         self.logger = logger.getChild(f'{self.__class__.__name__}')
         self.attr1 = "attr1"
         self.attr2 = 2
-    
+
     def my_method(self):
         do_something()
         self.logger.info('Doing something!')
 ```
 
 ## Test conventions
+
 - The same file hierarchy should be used between a project and the associated tests.
+
 ```
 my_project
     |my_project
@@ -71,59 +85,111 @@ my_project
     |   |test_my_module.py
     |   |
 ```
-- Please respect the matching: one class method for one class (or one function if there is no class). 
+
+- Please respect the matching: one class method for one class (or one function if there is no class).
 - Respect the following syntax (given when then & only one assert):
+
 ```python
 def my_function(arg):
     toto = ''
     if arg:
         toto += 'a string'
     return toto
 
+
 class TestMyFunction:
     def test_returns_a_string_if_arg_is_true(self):
         # Given
         arg = True
         expected = 'a string'
-    
+
         # When
         result = my_function(arg)
-    
+
         # Then
         assert result == expected
 
     def test_returns_empty_string_if_arg_is_none(self):
         # Given
         arg = None
         expected = ''
-    
+
         # When
         result = my_function(arg)
-    
+
         # Then
         assert result == expected
 ```
+
 - If a mock is needed, use a decorator instead of a context manager:
+
 ```python
 def my_function(arg):
     another_function(arg)
 
+
 class TestMyFunction:
     @patch('aivi.path.to.module.another_function')
     def test_returns_a_string_if_arg_is_true(self, mock_another_function):
         # Given
         arg = True
-    
+
         # When
         my_function(arg)
-    
+
         # Then
         mock_another_function.assert_called_once_with(arg)
 ```
-- Don't mistake a stub for a mock. A mock is used to assert that it has been called (see above example). A stub 
-is used to simulate the returned value.
-
+
+- Don't mistake a stub for a mock. A mock is used to assert that it has been called (see above example). A stub
+  is used to simulate the returned value.
+
+### Testing and docker images
+
+- In order to run the tests, your docker instance will need to be connected to GitHub, allowing docker to pull the
+  images.
+  Duplicate the `VIO/edge_orchestrator/.env.template` file, rename it `VIO/edge_orchestrator/.env` and fill it with your
+  GitHub username and access token, to make this connection possible. The token needs `read:packages`, `write:packages`
+  and `delete:packages` permissions.
+  [More informathion here](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-with-a-personal-access-token-classic)
+
+⚠ If you are not working with a M1 processor ⚠
+
+You may encounter some deployment problems when starting the Orchestrator's tests.
+To resolve them you will have to build a docker image that fits your system using the Orchestrator's makefile and modify
+the `conftest.py` file to edit the `image_name` field.
+
+```
+VIO/edge_orchestrator/tests/conftest.py
+
+EDGE_MODEL_SERVING = {
+    "image_name": --NEW_DOCKER_IMAGE--,
+    "container_volume_path": "/tf_serving",
+    "host_volume_path_suffix": "edge_model_serving",
+}
+```
+
+You may need to change the `starting_log` parameter and remove the call
+to `check_image_presence_or_pull_it_from_registry`
+from the `container.py` file.
+
+```
+VIO/edge_orchestrator/tests/fixtures/containers.py
+
+if tf_serving_host is None or tf_serving_port is None:
+    port_to_expose = 8501
+    container = TfServingContainer(
+        image=image_name,
+        port_to_expose=port_to_expose,
+        env={"MODEL_NAME": exposed_model_name},
+        host_volume_path=host_volume_path,
+        container_volume_path=container_volume_path,
+    )
+    container.start("INFO:     Application startup complete.")
+```
+
 ## Versioning strategy
+
 - Git tutorial:
     - [Basic git tutorial](http://rogerdudler.github.io/git-guide/)
     - [Learn git branching](https://learngitbranching.js.org/)
@@ -136,7 +202,7 @@ is used to simulate the returned value.
     - _X_ needs to be incremented each time a __major__ change is made (e.g. Python 2 to Python 3)
     - _Y_ needs to be incremented each time a __minor__ change is made (e.g. new features)
     - _Z_ needs to be incremented each time a __patch__ is made (e.g. bug fixing)
-- Versioning strategy: [Trunk Based Development](https://trunkbaseddevelopment.com/). 
+- Versioning strategy: [Trunk Based Development](https://trunkbaseddevelopment.com/).
   To implement a new US, please follow the steps:
     - Checkout on master and update the branch with the remote repository
     - Create a new branch following the naming convention below and checkout on it:
@@ -146,28 +212,35 @@ is used to simulate the returned value.
     - Make sure that the `README.md` doesn't need to be updated
     - Add the files to be tracked and commit your modifications
     - Open a merge request with target `master` (check the `squach commits` and `remove branch after merge` options)
-    - The MR must launch a CI pipeline (lint, test and build stage) and this pipeline needs to be green to allow to merge the MR
+    - The MR must launch a CI pipeline (lint, test and build stage) and this pipeline needs to be green to allow to
+      merge the MR
     - The final MR commit must follow the following naming conventing: `[US-ID] Add my brand new feature`
     - Organize a review with all your teammates to challenge and validate the code
     - All discussions must be closed before merging
-    - After validation: rebase your branch on master, wait for the CI pipeline to be green and merge on master (`fast-forward` option activated on gitlab)
+    - After validation: rebase your branch on master, wait for the CI pipeline to be green and merge on
+      master (`fast-forward` option activated on gitlab)
 
 Reminder of the git commands:
 
 git checkout master && git fetch -p origin && git reset --hard origin master
 git checkout -b 123_add_a_new_feature
-### 
+
+###  
+
 git add my_file.py
 git commit -m '[US-ID] Add my brand new feature'
 git push origin 123_add_a_new_feature
+
 ###
+
 git checkout master
 git pull --rebase origin/master
 pytest tests
 
 git tag (Listing the existing tags)
 git tag -a <version> -m <message> (Create new tag)
 git push origin <version> (Sharing the tags to the remote)
+
 ```
 - When to do?
 	- When you want to add new dependency package.