Skip to content

Latest commit

 

History

History
112 lines (79 loc) · 5.66 KB

README_DEV.md

File metadata and controls

112 lines (79 loc) · 5.66 KB

Readme for developers and power users

Most people will probably just use the feeds and bot I provide, so there is no need to clutter the readme with information only useful for developers or power users that want to self-host. So this goes here! It's currently mostly a quick braindump for my convenience. Often used commands, concepts and tasks. It is guaranteed to not be complete. At all.

Run this locally

So if you actually want to run this yourself, you first need an installed Python 3.12+ environment. The commands below are for Windows. If you use Linux, you probably know the right substitutes.

  • Download repository
  • Install poetry (https://python-poetry.org/)
  • Create virtual environment (python -m venv .venv)
  • Activate virtual environment (./.venv/Scripts/Activate)
  • Install the required packages (poetry install)
  • Install playwright browser (playwright install chrome)
  • Run (poetry run lootscraper)

Settings

On the first startup, a default configuration file will be created in ./data/config.toml. You can edit this file to change the settings (e.g. the sites to visit and the actions to perform). It also stores the needed API keys (see below), so just put them in there and then start the application again.

IGDB API Keys

See https://api-docs.igdb.com/#about for details, how to get an application ID and secret for the IGDB API.

Telegram Bot

Start a chat with Botfather, send him the /create command and follow the guided process. You will get an access token from him to put in the settings file. To get your user ID or the id of the group chat you are in, just send /debug to my public Telegram LootScraperBot. You can put that into "DeveloperChatId" config entry. Telegram related errors will be sent there.

Setting up the development environment

In addition to the above steps ("Run this locally"), I recommend installing the additional dev packages (poetry install --with dev) and the recommended VS Code extensions (Extensions > Filter > Recommended).

Build and run as Docker container

Docker needs to be downloaded and installed first. If you want to skip the build step, you can use https://hub.docker.com/r/eikowagenknecht/lootscraper as the image. Use the "main" tag to get the latest build from this repository.

  • Download repository
  • In terminal go to directory
  • First run:
    • Build: docker build . -t eikowagenknecht/lootscraper:develop
    • Start: docker run --detach --volume /your/local/path/to/data:/data --name lootscraper eikowagenknecht/lootscraper:develop
  • Update:
    • Stop: docker stop lootscraper
    • Remove: docker container rm lootscraper
    • Build without cache: docker build . --no-cache -t eikowagenknecht/lootscraper:develop
    • Start: docker run --detach --volume /your/local/path:/data --name lootscraper eikowagenknecht/lootscraper:develop
  • Push: docker push eikowagenknecht/lootscraper:develop
  • Save locally: docker image save -o ./data/lootscraper.tar eikowagenknecht/lootscraper:develop
  • Debug: docker run -it --entrypoint /bin/bash --volume /your/local/path:/data --name lootscraper_debug eikowagenknecht/lootscraper:develop
  • To stop, run docker stop lootscraper

SQLAlchemy

Overview of modern ORM statements (e.g. select).

Alembic

New database revisions are created in the following way:

  1. Update the metadata / ORM model
  2. Create a candidate: alembic revision --autogenerate -m "Fancy revision description"
  3. Check and edit the candidate
  4. Upgrade the database: alembic upgrade head

Some quick hints:

Poetry

Install: poetry install --with dev Install and remove unused: poetry install --with dev --sync Build: poetry build Add package: poetry add <package> Add dev package: poetry add <package> --group dev Run: poetry run <package>

Update dependencies

Update poetry itself:

  • (Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | py - --uninstall
  • (Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | py -

Install tool: poetry self add poetry-plugin-up (only once) Update all dependencies in pyproject.toml: poetry up --latest --with dev Install new dependencies locally: poetry install --with dev --sync Check if dependencies are up to date: poetry show --outdated --all

Profiling

For profiling, run scalene -m lootscraper.

Contribute

  • If you want to contribute, please open an issue first to discuss the changes.
  • Please follow the Conventional Commits specification for commit messages. The following types can be used:
    • feat: A new feature (minor version bump)
    • fix: A bug fix (patch version bump)
    • refactor: A code change that neither fixes a bug nor adds a feature
    • style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
    • docs: Documentation only changes
    • perf: A code change that improves performance
    • test: Adding missing or correcting existing tests
    • chore: Maintenance tasks, dependency updates and other non-user-facing changes
    • ci: Changes to the CI configuration files and scripts (GitHub Actions)
    • revert: Reverts a previous commit. In the body, it should say: This reverts commit <hash>.
  • Run ruff (poetry run ruff check . --fix) and ruff format (poetry run ruff format .) before committing.