Skip to content

Setting up a development environment

Jump to bottom
Jonathan Stray edited this page Sep 13, 2018 · 51 revisions

One-time Docker setup

  1. Install Docker. We use Docker because it lets us develop cross-platform.
  2. On Mac or Windows, increase Docker's memory to 4GB. (You don't need this step on Linux.) Mac instructions Windows instructions
  3. Install Git. That's XCode on Mac, GitHub Desktop for Windows, and dnf install git / apt install git on Fedora/Ubuntu Linux.
  4. git clone https://github.com/CJWorkbench/cjworkbench.git to clone this repo

Developer workflow

You'll need a command line. First, get the development server up and running

  1. git pull
  2. bin/dev start to run database migrations, start compiling JavaScript/CSS files, and run Django.
  3. Browse to http://localhost:8000 and register a user.
  4. Django will fake-email you a confirmation code. Check the local_mail/ subdirectory for the email and follow the link within. Then log in.
  5. Press Ctrl+C in the console to kill everything

Handy commands

  1. bin/dev python -- open a Python console with Workbench's dependencies installed. This is useful for testing Pandas-related code (e.g., import pandas works).
  2. bin/dev python ./manage.py shell -- open a Python console with Django loaded. This is useful for testing Workbench models (e.g., import server.models works).
  3. bin/dev sql -- open a PostgreSQL shell to the development database.
  4. bin/dev npm install PACKAGE -- you may run npm commands that modify package.json and package-lock.json; modules are installed to a Docker volume, not ./node_modules.
  5. bin/dev pipenv install PACKAGE -- you may run pipenv commands that modify Pipfile and Pipfile.lock; packages are installed to a Docker volume.
  6. bin/dev clean -- delete all Python packages, Node modules, the database, and files on disk that the database references.

Running tests

  • bin/dev npm test (~15s) runs JavaScript tests in assets/js/. bin/dev npm test -- --watch watches them for changes.
  • bin/dev unittest (~90s) runs Django tests in server/tests/. bin/dev unittest -k server.tests.modules.test_loadurl (~30s first run, ~6s subsequent runs) specifies just one module to test.
  • bin/integration-test (~15min) builds images exactly as we do in production and runs automated integration tests on them. It fails as soon as a single test fails, to save you time. Write tests in integrationtests/*.py; if there's an error, bin/integration-test will fill your screen with helpful advice.

If you test features that send email (such as confirmation email for new accounts) the email will appear in the local_mail folder in your cjworkbench directory.

Using PyCharm

To develop with the integrated tools in PyCharm (run, test, debug, etc.) setup a Python interpreter using the docker-compose.yml file, as described in this documentation.

You will need to tell run the python interpreter inside the frontend container, and launch it with the pipenv-run-python script to make sure it uses the correct virtual environment. This is what your PyCharm Python interpreter settings should look like:

PyCharm docker setup

On Mac, you can get to this dialog through PyCharm -> Preferences -> Project: cjworkbench -> Project Interpreter then pick Show All at the bottom of the Project Interpreter drop-down.

To test social account logins

If you want to test logging in with Facebook and Google, you'll need to either manually set up social applications in the Django admin interface, or get or create a file called socialaccounts_secrets.json in the project root. The file contains a list of objects defining a social account, like so:

{
    'name':'Display Name',
    'provider':'social_acccount_provider', // a full list is here: https://django-allauth.readthedocs.io/en/latest/providers.html
    'client_id':'12345', // Your chosen provider's app id/client id -- everyone calls this something different
    'secret':'12345' // Same as above but for client_secret
}

Once you've created this file, kill and restart bin/dev start.

To develop certain features

Google OAuth

To download from Google Drive:

  1. Browse to https://console.developers.google.com
  2. Create a new organization, if wanted
  3. Create a new project within that organization: Workbench Dev
  4. Enable the Google Drive API
  5. Enable the Google Picker API
  6. Add credentials for Google Drive API for a Web server accessing User data: Name GoogleConnect, authorized origin http://localhost:8000, authorized redirect URI http://localhost:8000/oauth, product name Workbench Dev.
  7. Download client_id.json or client_secret_XXXXX.json (they're the same)
  8. Move that file into your Workbench project directory, as client_secret.json. (It's in .gitignore: Google terms dictate it stay private.)
  9. Restart bin/dev start
  10. Create a new workflow and add a Google Drive module.
  11. Sign in.
  12. Pick a file.
  13. Click Update

Twitter OAuth

To let users authenticate with Twitter:

  1. Browse to https://apps.twitter.com
  2. Click Create New App. Call it Workbench Dev [random numbers], description Authenticator for CJWorkbench Twitter module, in dev mode only, website http://example.org, callback URL http://localhost:8000/oauth. Read and agree to the developer agreement. Click Create your Twitter application.
  3. Browse to Permissions. Switch to Read only and click Update Settings.
  4. Browse to Keys and Access Tokens. Copy/paste these into a new file called twitter_secret.json in the root of the project (we .gitignore it, because Twitter asks us to keep it private) with contents {"key":"....","secret":".....","redirect_url":"http://localhost:8000/oauth"}.
  5. Restart bin/dev start
  6. Create a new workflow and add a Twitter module.
  7. Sign in.
  8. Pick a search.
  9. Click Update
Clone this wiki locally