Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use a freshly baked bun #713

Merged
merged 45 commits into from
Apr 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
4bfe171
Install `bun` instead of `node` in dev container
lukasjuhrich Apr 13, 2024
701ceb9
Correctly interpret bun lockfiles
lukasjuhrich Apr 13, 2024
ff48a3d
Replace npm by bun
lukasjuhrich Apr 13, 2024
3c764b0
Update webpack cli to v5
lukasjuhrich Apr 13, 2024
7d9f54f
Add missing dependencies and trust
lukasjuhrich Apr 13, 2024
32001dd
Add `npm-check-updates`
lukasjuhrich Apr 13, 2024
d2106b3
docker: Remove unused and broken debug info from webpack command
lukasjuhrich Apr 13, 2024
e8d751a
ci: replace npm outdated script by bun alternative
lukasjuhrich Apr 13, 2024
2210e37
Install `dev` deps in prod builder as well
lukasjuhrich Apr 13, 2024
2359adf
ci: Remove progress bars from output
lukasjuhrich Apr 14, 2024
ac69246
ci: Add helpful debug output to `test` pipeline
lukasjuhrich Apr 14, 2024
7f41cd0
update some major webpack dependencies
lukasjuhrich Apr 14, 2024
7512a94
install esbuild
lukasjuhrich Apr 18, 2024
e6df9be
add initial esbuild script
lukasjuhrich Apr 19, 2024
ccf7a22
use glob (`fast-glob`) in esbuild script
lukasjuhrich Apr 19, 2024
84ee8f1
First kind-of-working esbuild.jsm config
lukasjuhrich Apr 19, 2024
408cb36
upgrade d3 to v7 (and see what breaks)
lukasjuhrich Apr 19, 2024
dcb9a13
Remove nvd3 and replace module usage by stub
lukasjuhrich Apr 20, 2024
e43bcc1
Update many d3 usages to v7
lukasjuhrich Apr 20, 2024
b7bece9
move assets to different bundle and switch to ESM
lukasjuhrich Apr 20, 2024
80144f7
Canonicalize entrypoints in manifest
lukasjuhrich Apr 20, 2024
665846b
expose `table` module to `window` object
lukasjuhrich Apr 20, 2024
826e8ae
cosmetic changes (and fix `catch` position)
lukasjuhrich Apr 20, 2024
b2b8c7e
Stop double import of `balance-chart`
lukasjuhrich Apr 20, 2024
a45864f
Fix axes rendering of balance chart
lukasjuhrich Apr 27, 2024
bba746d
Turn metafile generation into plugin
lukasjuhrich Apr 27, 2024
a4d40c9
Extract build options and add a hard-coded watch mode
lukasjuhrich Apr 27, 2024
a0baf6b
Add `--watch` and `--prod` flags to `esbuild.jsm` file
lukasjuhrich Apr 27, 2024
794823c
Turn bundling script into a typescript file
lukasjuhrich Apr 27, 2024
96de789
Allow to handle chunks during manifest generation
lukasjuhrich Apr 27, 2024
cdde0e2
Clean up destination directory when bundling with `--prod`
lukasjuhrich Apr 27, 2024
d695f61
Remove `javascrript-time-ago` import without side effects
lukasjuhrich Apr 27, 2024
28f548f
Include `esbuild-plugin-clean` to report build time in watch mode
lukasjuhrich Apr 27, 2024
64ea028
Add new `package.json` scripts for use with `bun`
lukasjuhrich Apr 27, 2024
12f3647
Include tree shaking in prod mode
lukasjuhrich Apr 27, 2024
44112d2
Install apexcharts
lukasjuhrich Apr 27, 2024
a343811
Replace traffic chart by apexchart alternative
lukasjuhrich Apr 28, 2024
b117bd9
Import scripts as ESM by default
lukasjuhrich Apr 28, 2024
e5ec81b
Fix CI
lukasjuhrich Apr 28, 2024
d24d50b
Replace `dev-webpack` by `dev-bundler`
lukasjuhrich Apr 28, 2024
388ff78
Specify proper JS/TS indentation
lukasjuhrich Apr 28, 2024
2371530
Remove last traces of webpack
lukasjuhrich Apr 28, 2024
3459084
Merge branch 'esbuild' into bun
lukasjuhrich Apr 28, 2024
7f32ab7
remove other superfluous js packages
lukasjuhrich Apr 29, 2024
8a12c38
update semgrep to v1.70.0
lukasjuhrich Apr 29, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ tab_width = 2
indent_size = 4
tab_width = 4

[{*.ts,*.tsx,*.tsm,*.js,*.jsx,*.jsm}]
indent_size = 4
tab_width = 4

[{*.py,*.pyi}]
indent_style = space
indent_size = 4
Expand Down
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
requirements.*.txt linguist-generated=true
*.lockb binary diff=lockb
20 changes: 10 additions & 10 deletions .github/workflows/docker-image.yml
Original file line number Diff line number Diff line change
Expand Up @@ -103,15 +103,13 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
- uses: oven-sh/setup-bun@v1
with:
node-version: 16
- run: npm ci
- run: npm run build
bun-version: 1.1.3
- run: bun install --frozen-lockfile
- run: bun run bundle --prod
- name: Check for outdated NPM packages
run: |
(npm outdated --json | tee outdated.json) || [[ $? == 1 ]]
python ./scripts/render_outdated.py outdated.json >> "${GITHUB_STEP_SUMMARY}"
run: bun x npm-check-updates --root --format group >> "${GITHUB_STEP_SUMMARY}"
build:
runs-on: ubuntu-latest
steps:
Expand Down Expand Up @@ -159,9 +157,11 @@ jobs:
- name: Run test-app entrypoints
run: docker compose -f docker-compose.test.yml run --rm --no-deps test-app
- name: Start
run: docker compose -f docker-compose.test.yml up --wait --wait-timeout=60 test-app
- name: Run webpack
run: docker compose -f docker-compose.test.yml run --rm test-app webpack --mode production
run: docker compose -f docker-compose.test.yml up --quiet-pull --wait --wait-timeout=60 test-app
- name: list npm packages for debug output
run: docker compose -f docker-compose.test.yml run --rm test-app shell bun pm ls --all
- name: Run esbuild
run: docker compose -f docker-compose.test.yml run --rm test-app bun run bundle --prod
- name: Run tests
run: >
docker compose -f docker-compose.test.yml run --rm test-app
Expand Down
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ repos:
args: [pyproject.toml, --generate-hashes, --extra, prod, --quiet, -o, requirements.prod.txt]
files: ^(pyproject.toml|requirements.prod.txt)$
- repo: https://github.com/returntocorp/semgrep
rev: 'v1.31.2'
rev: 'v1.70.0'
hooks:
- id: semgrep
# See semgrep.dev/rulesets to select a ruleset and copy its URL
Expand Down
Binary file added bun.lockb
Binary file not shown.
139 changes: 139 additions & 0 deletions bundle.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
#!/usr/bin/env bun
import { parseArgs } from "util";
import * as esbuild from 'esbuild';
import { clean } from 'esbuild-plugin-clean';
import { summaryPlugin } from 'esbuild-plugin-summary';
import fs from 'fs';
import path from 'path';

const src = path.resolve(__dirname, 'web', 'resources');
const dst = path.resolve(__dirname, 'web', 'static');

const entryPoints = [
['advanced-search', './js/advanced-search.js'],
['balance-chart', './js/balance-chart.js'],
['lazy-load-select', './js/lazy-load-select.js'],
['mac-address-input', './js/mac-address-input.js'],
['confirmable-error', './js/confirmable-error.ts'],
['navigation', './js/navigation.js'],
['tooltip', './js/tooltip.js'],
['table-fixed-header', './js/table-fixed-header.js'],
['traffic-graph', './js/traffic-graph.js'],
['transaction-chart', './js/transaction-chart.js'],
['transaction-form', './js/transaction-form.js'],
['unlimited-end-date', './js/unlimited-end-date.js'],
['select-multiple-improvement', './js/select-multiple-improvement.js'],
['tab-anchor', './js/tab-anchor.ts'],
['user-suite', './js/user-suite.ts'],
['rooms-table', './js/rooms-table.ts'],
].map((x) => (
{out: x[0], in: path.join(src, x[1])}
))

const { values: args } = parseArgs({
args: Bun.argv,
options: {
watch: { type: 'boolean' },
prod: { type: 'boolean' },
},
strict: true,
allowPositionals: true,
})

const options: esbuild.BuildOptions = {
logLevel: "info",

bundle: true,
splitting: args.prod,
minify: args.prod,
treeShaking: args.prod,
sourcemap: true,
target: 'es2016',
format: 'esm',
metafile: true,
absWorkingDir: src,

entryPoints: [
{in: `${src}/main.js`, out: 'main'},
{in: `${src}/assets.js`, out: 'assets'},
...entryPoints,
],
outdir: dst,
assetNames: '[dir]/[name].[hash]',
chunkNames: 'chunks/[name].[hash]',
entryNames: '[name].[hash]',

loader: {
'.xml': 'copy',
'.png': 'copy',
'.ico': 'copy',
'.svg': 'copy'
},

inject: [path.join(src, "inject-jquery.js")],
plugins: [{name: "manifest", setup(build){
build.onEnd(result => generateManifest(result, src, dst))}
}].concat(args.prod ? [
clean({
patterns: [dst + "/**"],
})
] : [])
.concat(args.watch ? [
summaryPlugin()
] : [])
}

if (args.watch) {
const ctx = await esbuild.context(options)
await ctx.watch()
} else {
esbuild.build(options)
}


// for debug purposes
// fs.writeFileSync(path.join(dst, "meta.json"), JSON.stringify(result.metafile))

// cleanup jobs: turn `result.metafile` into manifest.
function generateManifest(result, src, dst) {
const outs = result.metafile.outputs

let entries = {}
for (const [outname_, outprops] of Object.entries(outs)) {
const outname = rebase(outname_, src, dst)
const entry = deriveEntryName(outname, outprops)
if (entry !== undefined) {
entries[entry] = outname
}
}
fs.writeFileSync(path.join(dst, "manifest.json"), JSON.stringify(entries))
}

function deriveEntryName(outname, outprops) {
const {entryPoint, inputs: inputs_} = outprops
const inputs = inputs_ !== undefined ? Object.keys(inputs_) : undefined
if (outname.endsWith('.map')) {
return null
}
if (entryPoint !== undefined) {
return entryPoint.replace(/^js\//, "").replace(/\.ts$/, ".js")
}
else if (outname.match(/main\..*\.css/)) {
return "main.css"
}
else if (inputs !== undefined && inputs.length == 1) {
return inputs[0]
}
else if (!outname.includes("chunk")) {
throw Error(`output asset ${outname} has no entrypoint and has no unique input (inputs: ${inputs})`)
}
}

/**
* Given a path relative to `base_old`, derive a presentation relative to `base_new`.
*/
function rebase(p, base_old, base_new) {
const abs = path.normalize(path.join(base_old, p));
return path.relative(base_new, abs)
}

4 changes: 2 additions & 2 deletions doc/guides/database.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ The password for the ``postgres`` user is ``password``.
.. code:: shell

# clone the anonymized dump into `data/`
docker compose stop dev-app dev-webpack
docker compose stop dev-app dev-bundler
git clone https://git.agdsn.de/AGDSN/pycroft-data.git data --depth=1
export PGPASSFILE=.pycroft.pgpass
psql -wb postgres://[email protected]:55432/pycroft \
Expand All @@ -28,7 +28,7 @@ The password for the ``postgres`` user is ``password``.
-f data/pycroft_schema.sql \
-f data/pycroft.sql
# start the web app again
docker compose start dev-app dev-webpack
docker compose start dev-app dev-bundler

Success
Navigate to `<http://localhost:5000>`_.
Expand Down
6 changes: 3 additions & 3 deletions doc/guides/troubleshooting.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,12 @@ Due to laziness, prefix every bash-snippet with
Webpack appears to be missing a library
---------------------------------------

Re-Install everything using npm, and re-run the webpack entrypoint.
Re-Install everything using npm, and restart the bundling

.. code:: sh

drc run --rm dev-app shell npm ci
drc run --rm dev-app webpack
drc run --rm dev-app bun i --frozen-lockfile
drc run --rm dev-app bun run bundle

Pip appears to be missing a dependency
--------------------------------------
Expand Down
6 changes: 3 additions & 3 deletions docker-compose.dev.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ services:
condition: service_healthy
dev-mq:
condition: service_healthy
dev-webpack:
dev-bundler:
condition: service_started
networks:
dev:
Expand All @@ -30,11 +30,11 @@ services:
test: curl --fail http://localhost:5000
interval: 2s
start_period: 10s
dev-webpack:
dev-bundler:
extends:
file: docker-compose.base.yml
service: dev
command: ["webpack", "--watch"]
command: ["bun", "run", "bundle", "--watch"]
dev-docs:
extends:
file: docker-compose.base.yml
Expand Down
17 changes: 14 additions & 3 deletions docker/dev.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,7 +1,19 @@
# syntax=docker/dockerfile:1.4
# syntax=docker/dockerfile:1.6
# Copyright (c) 2015 The Pycroft Authors. See the AUTHORS file.
# This file is part of the Pycroft project and licensed under the terms of
# the Apache License, Version 2.0. See the LICENSE file for details.
FROM alpine:latest AS bunzipper
RUN --mount=type=cache,target=/var/cache/apk,sharing=locked \
apk add unzip curl
RUN <<EOF ash
set -euo pipefail
curl -sSfLO https://github.com/oven-sh/bun/releases/download/bun-v1.1.3/bun-linux-x64-baseline.zip
unzip -j bun-linux-x64-baseline.zip bun-linux-x64-baseline/bun -d /opt
echo "e1c94765691f95ca593cf921c89d7bba951cd6e876d28f67ee37a3feeb288f55 /opt/bun" \
| sha256sum -c -
EOF

# syntax=docker/dockerfile:1.4
FROM pycroft-base

USER root
Expand All @@ -23,13 +35,12 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
git \
less \
libpq-dev \
nodejs \
npm \
postgresql-client \
strace \
unzip \
vim \
&& apt-get clean
COPY --chmod=755 --from=bunzipper /opt/bun /usr/local/bin/bun

COPY --link . /
COPY --link --chmod=755 ./container /container
Expand Down
17 changes: 17 additions & 0 deletions docker/dev/container/commands/bun
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
#!/bin/bash

set -euo pipefail

readonly USAGE=("[args]")

readonly DESCRIPTION=(
"Run bun in project root. Optional arguments are passed to bun."
)

run() {
set -x
cd /opt/pycroft/app
exec bun "$@"
}

[[ "$0" == "$BASH_SOURCE" ]] && run "$@" || :
33 changes: 0 additions & 33 deletions docker/dev/container/commands/webpack

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,14 @@ LOCKDIR=/opt/pycroft/app/node_modules
. /container/common/locking.sh

function execute_hook() {
set -x
cd /opt/pycroft/app

if shopt -s nullglob dotglob; files=(node_modules/*); (( ${#files[@]} > 0 )); then
return
fi

npm ci
bun install --frozen-lockfile
}

[[ -d $LOCKDIR ]] || mkdir $LOCKDIR
Expand Down
7 changes: 3 additions & 4 deletions docker/prod.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,12 @@ RUN /opt/pycroft/venv/bin/pip wheel --wheel-dir /opt/pycroft/wheel -r requiremen
&& /opt/pycroft/venv/bin/pip wheel --wheel-dir /opt/pycroft/wheel --no-deps ./deps/wtforms-widgets

# Download JS/CSS dependencies
COPY --chown=pycroft:pycroft package.json package-lock.json ./
RUN npm ci \
npm outdated
COPY --chown=pycroft:pycroft package.json bun.lockb ./
RUN bun install --frozen-lockfile

# Build Pycroft wheel
COPY --chown=pycroft:pycroft . .
RUN npm run --prod build
RUN bun run bundle --prod
RUN /opt/pycroft/venv/bin/pip wheel --no-deps --wheel-dir /opt/pycroft/wheel .

FROM pycroft-base
Expand Down
Loading
Loading