Merge pull request #500 from uktrade/release/slivovitz

Release Slivovitz

README.md

@@ -1,3 +1,5 @@
+# Data Hub Leeloo
+
 [![image](https://circleci.com/gh/uktrade/data-hub-leeloo/tree/master.svg?style=svg)](https://circleci.com/gh/uktrade/data-hub-leeloo/tree/master)
 [![image](https://codecov.io/gh/uktrade/data-hub-leeloo/branch/master/graph/badge.svg)](https://codecov.io/gh/uktrade/data-hub-leeloo)
 [![image](https://codeclimate.com/github/uktrade/data-hub-leeloo/badges/gpa.svg)](https://codeclimate.com/github/uktrade/data-hub-leeloo)
@@ -6,8 +8,7 @@
 
 Leeloo provides an API into Data Hub for Data Hub clients. Using Leeloo you can search for entities and manage companies, contacts and interactions.
 
-Installation with Docker
-========================
+## Installation with Docker
 
 Leeloo uses Docker compose to setup and run all the necessary components. The docker-compose.yml file provided is meant to be used for running tests and development.
 
@@ -27,23 +28,23 @@ Leeloo uses Docker compose to setup and run all the necessary components. The do
 3.  Populate the db:
 
     ```shell
-    docker-compose run leeloo python manage.py migrate
-    docker-compose run leeloo python manage.py loadmetadata
-    docker-compose run leeloo python manage.py load_omis_metadata
-    docker-compose run leeloo python manage.py createinitialrevisions
+    docker-compose run leeloo ./manage.py migrate
+    docker-compose run leeloo ./manage.py loadmetadata
+    docker-compose run leeloo ./manage.py load_omis_metadata
+    docker-compose run leeloo ./manage.py createinitialrevisions
     ```
 4. Optionally, you can load some test data and update elasticsearch:
 
     ```shell
-    docker-compose run leeloo python manage.py loaddata /app/fixtures/test_data.yaml
+    docker-compose run leeloo ./manage.py loaddata /app/fixtures/test_data.yaml
 
-    docker-compose run leeloo python manage.py sync_es
+    docker-compose run leeloo ./manage.py sync_es
     ```
 
 5.  Create a superuser:
 
     ```shell
-    docker-compose run leeloo python manage.py createsuperuser
+    docker-compose run leeloo ./manage.py createsuperuser
     ```
 
 6.  Run the services:
@@ -52,36 +53,12 @@ Leeloo uses Docker compose to setup and run all the necessary components. The do
     docker-compose up
     ```
 
-7.  To set up the [data hub frontend app](https://github.com/uktrade/data-hub-frontend), log into the [django admin](http://localhost:8000/admin/oauth2_provider/application/) and add a new oauth application with:
-
-        - Client type: Confidential
-        - Authorization grant type: Resource owner password-based
-
-8.  Add the client id / client secret to the frontend .env file
-
-Local development with Docker
------------------------------
-
-To run the tests:
-
-```shell
-docker-compose run leeloo bash tests.sh
-docker-compose run leeloo bash tests-auth.sh
-```
-
-To run the linter:
-
-```shell
-docker-compose run leeloo flake8
-```
-
-Native installation (without Docker)
-====================================
+## Native installation (without Docker)
 
 Dependencies:
 
--   Python 3.6.1
--   Postgres (tested on 9.5+)
+-   Python 3.6.x
+-   PostgreSQL (tested on 9.5 and 9.6)
 
 1.  Clone the repository:
 
@@ -124,7 +101,7 @@ Dependencies:
     create database datahub;
     ```
 
-8. Make sure you have elasticsearch running locally. If you don't, you can run one in docker:
+8. Make sure you have Elasticsearch running locally. If you don't, you can run one in Docker:
 
     ```shell
     docker run -p 9200:9200 -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" elasticsearch:2.3
@@ -141,7 +118,7 @@ Dependencies:
     ./manage.py createinitialrevisions
     ```
 
-10. Optionally, you can load some test data and update elasticsearch:
+10. Optionally, you can load some test data and update Elasticsearch:
 
     ```shell
     ./manage.py loaddata fixtures/test_data.yaml
@@ -155,15 +132,9 @@ Dependencies:
     ./manage.py runserver
     ```
 
-12. To set up the [data hub frontend app](https://github.com/uktrade/data-hub-frontend), log into the [django admin](http://localhost:8000/admin/oauth2_provider/application/) and add a new oauth application with:
+## Local development
 
-        - Client type: Confidential
-        - Authorization grant type: Resource owner password-based
-
-13. Add the client id / client secret to the frontend .env file
-
-Local development (without Docker)
-----------------------------------
+If using Docker, prefix these commands with `docker-compose run leeloo`.
 
 To run the tests:
 
@@ -178,66 +149,133 @@ To run the linter:
 flake8
 ```
 
-Heroku
-======
+## Granting access to the front end
+
+To give access to the [internal front end](https://github.com/uktrade/data-hub-frontend):
+
+1. Log into the [Django admin applications page](http://localhost:8000/admin/oauth2_provider/application/) and add a new OAuth application with these details:
+
+    * Client type: Confidential
+    * Authorization grant type: Resource owner password-based
 
-Leeloo can run on any Heroku style platform. These environment variables MUST be configured:
+1. Define the required scopes for the app by adding a new record in the 
+[OAuth application scopes](http://localhost:8000/admin/oauth/oauthapplicationscope/) 
+page with these details:
+    * Application: The application just created
+    * Scope: `internal-front-end`
 
--   DATABASE\_URL
--   DATAHUB\_SECRET
--   DEBUG
--   DJANGO\_SECRET\_KEY
--   DJANGO\_SENTRY\_DSN
--   DJANGO\_SETTINGS\_MODULE
--   BULK\_CREATE\_BATCH\_SIZE (default=5000)
--   ES\_URL
--   ES\_INDEX
--   AWS\_ACCESS\_KEY\_ID
--   AWS\_SECRET\_ACCESS\_KEY
--   DOCUMENTS\_BUCKET
+1. Add the client ID and secret to the front-end environment variables
 
-Management commands
-===================
+## Granting access to machine-to-machine clients
 
+To give access to a machine-to-machine client that doesn't require user authentication: 
+
+1. Log into the [Django admin applications page](http://localhost:8000/admin/oauth2_provider/application/) and add a new OAuth application with these details:
+
+    * Client type: Confidential
+    * Authorization grant type: Client credentials
+
+1. Define the required scopes for the app by adding a new record in the 
+[OAuth application scopes](http://localhost:8000/admin/oauth/oauthapplicationscope/) 
+page with these details:
+    * Application: The application just created
+    * Scope: The required scopes
+
+The currently defined scopes can be found in [`datahub/oauth/scopes.py`](https://github.com/uktrade/data-hub-leeloo/tree/develop/datahub/oauth/scopes.py).
+
+[Further information about the available grant types can be found in the OAuthLib docs](http://oauthlib.readthedocs.io/en/stable/oauth2/grants/grants.html).
+
+## Deployment
+
+Leeloo can run on any Heroku-style platform. Configuration is performed via the following environment variables:
+
+
+| Variable name | Required | Description |
+| ------------- | ------------- | ------------- |
+| `AV_SERVICE_URL` | Yes | URL for ClamAV service. If not configured, virus scanning will fail. |
+| `AWS_ACCESS_KEY_ID` | No | Used as part of [boto3 auto-configuration](http://boto3.readthedocs.io/en/latest/guide/configuration.html#configuring-credentials). |
+| `AWS_DEFAULT_REGION` | No | [Default region used by boto3.](http://boto3.readthedocs.io/en/latest/guide/configuration.html#environment-variable-configuration) |
+| `AWS_SECRET_ACCESS_KEY` | No | Used as part of [boto3 auto-configuration](http://boto3.readthedocs.io/en/latest/guide/configuration.html#configuring-credentials). |
+| `BULK_CREATE_BATCH_SIZE`  | No | Used when loading Companies House records (default=5000). |
+| `CDMS_AUTH_URL`  | Yes | |
+| `DATABASE_URL`  | Yes | PostgreSQL server URL (with embedded credentials). |
+| `DATAHUB_FRONTEND_BASE_URL`  | Yes | |
+| `DATAHUB_SECRET`  | Yes | |
+| `DEBUG`  | Yes | Whether Django's debug mode should be enabled. |
+| `DJANGO_SECRET_KEY`  | Yes | |
+| `DJANGO_SENTRY_DSN`  | Yes | |
+| `DJANGO_SETTINGS_MODULE`  | Yes | |
+| `DOCUMENTS_BUCKET`  | Yes | S3 bucket for document storage. |
+| `ES_INDEX`  | Yes | |
+| `ES_URL`  | Yes | |
+| `ES_VERIFY_CERTS`  | No | |
+| `GUNICORN_ACCESSLOG`  | No | File to direct Gunicorn logs to (default=stdout). |
+| `GUNICORN_ACCESS_LOG_FORMAT`  | No |  |
+| `GUNICORN_WORKER_CLASS`  | No | [Type of Gunicorn worker.](http://docs.gunicorn.org/en/stable/settings.html#worker-class) Uses async workers via gevent by default. |
+| `GUNICORN_WORKER_CONNECTIONS`  | No | Maximum no. of connections for async workers (default=10). |
+| `OMIS_NOTIFICATION_ADMIN_EMAIL`  | Yes | |
+| `OMIS_NOTIFICATION_API_KEY`  | Yes | |
+| `OMIS_NOTIFICATION_OVERRIDE_RECIPIENT_EMAIL`  | No | |
+| `OMIS_PUBLIC_BASE_URL`  | Yes | |
+| `WEB_CONCURRENCY` | No | Number of Gunicorn workers (set automatically by Heroku, otherwise defaults to 1). |
+
+
+## Management commands
+
+If using Docker, remember to run these commands inside your container by prefixing them with `docker-compose run leeloo`.
+
+### CDMS authentication 
 Enable CDMS login for users (use this to let a CDMS user log in):
 
 ```shell
-docker-compose run leeloo python manage.py manageusers [email protected] [email protected] --enable
+./manage.py manageusers [email protected] [email protected] --enable
 ```
 
 Disable CDMS login for users:
 
 ```shell
-docker-compose run leeloo python manage.py manageusers [email protected] [email protected] --disable
+./manage.py manageusers [email protected] [email protected] --disable
 ```
+### Database
 
 Apply migrations:
 
 ```shell
-docker-compose run leeloo python manage.py migrate
+docker-compose run leeloo ./manage.py migrate
 ```
 
 If the database is freshly built or a new versioned model is added run:
 
 ```shell
-docker-compose run leeloo python manage.py createinitialrevisions
+./manage.py createinitialrevisions
 ```
 
 Load metadata:
 
 ```shell
-docker-compose run leeloo python manage.py loadmetadata
-docker-compose run leeloo python manage.py load_omis_metadata
+./manage.py loadmetadata
+./manage.py load_omis_metadata
+```
+
+### Elasticsearch
+
+Resync all Elasticsearch records:
+
+```shell
+./manage.py sync_es
 ```
 
-Update Elasticsearch:
+### Companies House
+
+Update Companies House records:
 
 ```shell
-docker-compose run leeloo python manage.py sync_es
+./manage.py sync_ch
 ```
 
-Dependencies
-============
+This truncates the Companies House table, downloads the latest data from Companies House and repopulates the table.
+
+## Dependencies
 
 Direct dependencies are specified in `requirements.in`. `requirements.txt` is a lock file generated using [pip-compile (from pip-tools)](https://github.com/jazzband/pip-tools) and should not be manually edited.
 

config/api_urls.py

@@ -40,5 +40,9 @@
     url(r'^', include((investment_urls, 'investment'), namespace='investment')),
     url(r'^', include((leads_urls, 'business-leads'), namespace='business-leads')),
     url(r'^', include((search_urls, 'search'), namespace='search')),
-    url(r'^omis/', include((omis_urls, 'omis'), namespace='omis'))
+    url(r'^omis/', include((omis_urls.internal_frontend_urls, 'omis'), namespace='omis')),
+    url(
+        r'^omis/public/',
+        include((omis_urls.public_urls, 'omis-public'), namespace='omis-public')
+    ),
 ]

config/settings/common.py

@@ -8,6 +8,7 @@
 """
 
 import environ
+
 from .companieshouse import *
 
 
@@ -57,13 +58,15 @@
     'datahub.investment',
     'datahub.leads',
     'datahub.metadata',
+    'datahub.oauth',
     'datahub.search.apps.SearchConfig',
     'datahub.user',
     'datahub.omis.core',
     'datahub.omis.order',
     'datahub.omis.market',
     'datahub.omis.notification',
     'datahub.omis.quote',
+    'datahub.omis.invoice',
 ]
 
 INSTALLED_APPS = DJANGO_APPS + THIRD_PARTY_APPS + LOCAL_APPS
@@ -135,6 +138,12 @@
     'django.contrib.auth.backends.ModelBackend'
 ]
 
+# django-oauth-toolkit settings
+
+OAUTH2_PROVIDER = {
+    'SCOPES_BACKEND_CLASS': 'datahub.oauth.scopes.ApplicationScopesBackend',
+}
+
 
 # Internationalization
 # https://docs.djangoproject.com/en/1.9/topics/i18n/
@@ -155,8 +164,12 @@
     'DEFAULT_VERSIONING_CLASS': 'rest_framework.versioning.NamespaceVersioning',
     'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.LimitOffsetPagination',
     'PAGE_SIZE': 100,
-    'DEFAULT_AUTHENTICATION_CLASSES': ['oauth2_provider.contrib.rest_framework.OAuth2Authentication'],
-    'DEFAULT_PERMISSION_CLASSES': ['rest_framework.permissions.IsAuthenticated'],
+    'DEFAULT_AUTHENTICATION_CLASSES': [
+        'oauth2_provider.contrib.rest_framework.OAuth2Authentication'
+    ],
+    'DEFAULT_PERMISSION_CLASSES': [
+        'oauth2_provider.contrib.rest_framework.IsAuthenticatedOrTokenHasScope'
+    ],
     'ORDERING_PARAM': 'sortby'
 }
 
@@ -186,6 +199,12 @@
 DATAHUB_FRONTEND_BASE_URL = env('DATAHUB_FRONTEND_BASE_URL', default='http://localhost:3000')
 
 # OMIS
+
+# if set, all the notifications will be sent to this address instead of the
+# intended recipient, useful for environments != live
+OMIS_NOTIFICATION_OVERRIDE_RECIPIENT_EMAIL = env(
+    'OMIS_NOTIFICATION_OVERRIDE_RECIPIENT_EMAIL', default=''
+)
 OMIS_NOTIFICATION_ADMIN_EMAIL = env('OMIS_NOTIFICATION_ADMIN_EMAIL', default='')
 OMIS_NOTIFICATION_API_KEY = env('OMIS_NOTIFICATION_API_KEY', default='')
 OMIS_NOTIFICATION_TEST_API_KEY = env('OMIS_NOTIFICATION_TEST_API_KEY', default='')

config/settings/test.py

@@ -21,5 +21,6 @@
 DOCUMENT_BUCKET='test-bucket'
 AV_SERVICE_URL='http://av-service/'
 
+OMIS_NOTIFICATION_OVERRIDE_RECIPIENT_EMAIL = ''
 OMIS_NOTIFICATION_ADMIN_EMAIL = '[email protected]'
 OMIS_NOTIFICATION_API_KEY = ''

conftest.py

@@ -1,3 +1,4 @@
+import factory
 import pytest
 from django.core.management import call_command
 from pytest_django.lazy_django import skip_if_no_django
@@ -11,6 +12,13 @@ def django_db_setup(django_db_setup, django_db_blocker):
         call_command('load_omis_metadata')
 
 
+@pytest.fixture(scope='session', autouse=True)
+def set_faker_locale():
+    """Sets the default locale for Faker."""
+    with factory.Faker.override_default_locale('en_GB'):
+        yield
+
+
 @pytest.fixture
 def api_request_factory():
     """Django REST framework ApiRequestFactory instance."""