Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce caching for dataframes #13

Draft
wants to merge 168 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
168 commits
Select commit Hold shift + click to select a range
66c854e
MALA inference runs, but new density format crashes the coord_array call
RandomDefaultUser Apr 4, 2023
a416320
MALA inference runs fully now
RandomDefaultUser Apr 5, 2023
25fe517
removed upload-component's fileformat-restriction; added 'UnknownFile…
Apr 7, 2023
aeddbfb
removed explicit type-check for .cube
Apr 7, 2023
23c43b7
updated for full inference-implementation
Apr 7, 2023
37ded86
removed old imports
Apr 9, 2023
9e0748f
input-temperature now cast to float before storing; now passing tempe…
Apr 9, 2023
83aac02
Added debug model (so debugging can be done without LAMMPS and QE)
RandomDefaultUser Apr 14, 2023
d9fab5a
Got rid of .cube file dependence for inference
RandomDefaultUser Apr 14, 2023
4c1f678
Changed atom limit to 200
RandomDefaultUser Apr 14, 2023
f2cbf16
Merge remote-tracking branch 'origin/mala_inference' into mala_inference
Apr 21, 2023
50881fa
atom list retracted by def. | settings sidebar auto-expanded on "run-…
Apr 21, 2023
614b38a
removed (hopefully) all traces of volume-plot | fixed outline checkbo…
Apr 26, 2023
80f7c9a
test-implementation of high-resolution rangeslider for x-axis-slicing
May 5, 2023
efcaff9
adjusting "pushable"-value of rangeslider, to achieve ~1-voxel-width …
Aug 4, 2023
f19fa9e
introduced models_list.json || fixed (new) x-cross section reset bug …
Aug 6, 2023
3e9866e
working on cell-preview after upload
Aug 6, 2023
b85f880
cell-preview needs work
Aug 9, 2023
3613bec
comment on cell-preview work
Aug 9, 2023
ea8e934
50% of cell-preview done
Aug 18, 2023
003c90e
visual loading progress after running inference properly implemented …
Aug 18, 2023
09f4a62
Atom-/Cell-Preview on upload is implemented
Aug 25, 2023
6424b20
removed hoverlines for atom/cell-preview
Aug 27, 2023
ac0ca9d
equalized methods for plot-crosssections; working on tooltips showing…
Aug 28, 2023
20a2d80
added TODOs
Aug 28, 2023
0c79ae4
added cell boundaries for main plot
Aug 29, 2023
c5151c7
1-layer slicing of the plot now works
Aug 29, 2023
efaa07d
ipmlementing Patch()-feature - settings need work
Aug 29, 2023
176d341
optimization, bug fixing and todos - outline checkbox bug reappeared,…
Aug 30, 2023
ee435e8
outline checkbox bug fixed again; loading spinner position fixed, oth…
Aug 30, 2023
b729c51
removed grid and axis for mainplot, removed grid for cell/ atom preview
Aug 30, 2023
a7c3efe
added slider-boundary-indicators for density 'slicing'
Aug 30, 2023
cad3326
Reduced atom size
RandomDefaultUser Sep 13, 2023
c0a7e6d
Merge pull request #1 from mala-project/mala_inference
RandomDefaultUser Sep 13, 2023
318ebe9
setup.py and requirements
Sep 29, 2023
806959d
setup.py and requirements
Sep 29, 2023
c9e17a4
refreshed setup.py | BUG from deprecated use of packaging.version.Leg…
Sep 29, 2023
feadf67
Reworked README.md
Oct 3, 2023
c7876ce
Tiny fix on README.md
Oct 3, 2023
4033f0f
Next tiny fix on README.md
Oct 3, 2023
11155cf
Next tiny fix on README.md
Oct 3, 2023
3dadadb
Reformatting and updating README.md
Oct 6, 2023
adffa36
Updates to README.md
Oct 6, 2023
59e86df
Including images in README.md
Oct 6, 2023
2c696dd
reorganized assets
Oct 6, 2023
e555dfe
removed .gitignore
Oct 6, 2023
3c2d6ed
Delete .gitignore
yyyxam Oct 6, 2023
9c4c3ee
Update README.md title
yyyxam Oct 6, 2023
844e437
requirements.txt: not as restrictive on newer versions (of some depen…
Oct 6, 2023
a72d39a
Merge remote-tracking branch 'origin/dev' into dev
Oct 6, 2023
99bde78
Fixed "edit" and "reset" buttons; still open to improvement
Oct 6, 2023
51750dc
less restrictive requirements.txt
Oct 9, 2023
191bda4
fix on bot-canvas-button not closing + idea for upload-modal not clos…
Oct 9, 2023
9cf885f
Update README.md
yyyxam Oct 16, 2023
e523915
Getting ready for release: changed default particle size; changed edi…
Oct 16, 2023
8953d72
Merge remote-tracking branch 'origin/dev' into dev
Oct 16, 2023
04b8e4f
Update README.md
yyyxam Oct 20, 2023
ef58a50
Update README.md
yyyxam Oct 20, 2023
1166842
restructuring files
Oct 20, 2023
39100e9
restructuring files
Oct 20, 2023
a5e63a7
using ase.Atoms.todict and fromdict to store Atoms-Obj, instead of re…
Oct 20, 2023
84f6a9b
added doc
Oct 20, 2023
fbe1fc4
auto-delete of uploaded file after it's read by ASE
Oct 20, 2023
eee4059
changed upload path in accordance to new project-structure
Oct 20, 2023
407de1b
cleaning up debug prints || added favicon, updated logos
Oct 29, 2023
975cea8
comments
Oct 29, 2023
b740956
introducing components for readability of app.py
Oct 31, 2023
8cd60d9
comments and fiddling with orientation-plot updates
Oct 31, 2023
73c295d
extracted buttons to their respective components
Oct 31, 2023
2ac402b
corrected image path for README.md
Oct 31, 2023
b7da729
corrected image path for README.md
Oct 31, 2023
4b51cf9
fixed (cached) logo
yyyxam Nov 13, 2023
6d53085
Refactored plot-tools into separate file | tried refactoring callback…
yyyxam Nov 20, 2023
36a173f
added dependencies for performance improvements
yyyxam Nov 24, 2023
e56ec32
simplyfied toggle_menu_offcanvas; renamed some components
yyyxam Dec 17, 2023
35fadd8
renamed settings and its button for clarity
yyyxam Dec 17, 2023
c22f929
Hotfix so that TEM is optional
RandomDefaultUser Dec 19, 2023
1b49fb6
Implemented functions to save inference output to
RandomDefaultUser Dec 28, 2023
62acf58
Fixed path in inference script
RandomDefaultUser Dec 28, 2023
7d6bdf2
Merge branch 'dev' of github.com:mala-project/malaWeb into dev
yyyxam Dec 28, 2023
fe188fa
refactored directory "models" | added model_paths.json |
yyyxam Dec 28, 2023
a08b119
Disabled "run-mala"-button after activation to prevent running MALA m…
yyyxam Dec 29, 2023
0db7147
Removed print statements
yyyxam Dec 29, 2023
1341f39
Fixed JS-console warnings, mostly for style name-conventions
yyyxam Dec 31, 2023
82c40ae
Introducing clientside callback for update_tools (wip) || removed pr…
yyyxam Dec 31, 2023
5496e99
working on clientside callback
yyyxam Jan 1, 2024
24b5314
update clientside callback for update_tools (wip)
yyyxam Jan 3, 2024
2e045e0
Fixed upload path
yyyxam Jan 4, 2024
3642de1
working on clientside callback
yyyxam Jan 4, 2024
5cc23ce
Merge remote-tracking branch 'origin/dev' into dev
yyyxam Jan 4, 2024
5b5beed
working external clientside_callback
yyyxam Jan 4, 2024
6c4569f
Merge remote-tracking branch 'origin/dev' into dev
yyyxam Jan 4, 2024
5f112f1
working on clientside callback
yyyxam Jan 4, 2024
5374102
rolling back update_tools changes
yyyxam Dec 31, 2023
4c21b9c
deleting temp files
yyyxam Jan 4, 2024
be829f4
cherry picked to reset update_tools changes
yyyxam Dec 31, 2023
026403c
cleaned up slider indicator updates
yyyxam Jan 4, 2024
0703551
Added session_id-parameter for saving .cube at correct location (expe…
yyyxam Jan 4, 2024
ca14f62
Added component and callback for downloading data created by inference
yyyxam Jan 4, 2024
8be4613
Disabled data-download after data-reset
yyyxam Jan 4, 2024
55495e3
Fixed bug of plot not updating after running inference on already upl…
yyyxam Jan 4, 2024
55a3a31
Changed default size options | added dcc.Download-component and buttons
yyyxam Jan 4, 2024
35fcfed
.gitignore updates
yyyxam Jan 5, 2024
27ee491
Merge branch 'dev' of github.com:mala-project/malaWeb into dev
yyyxam Jan 5, 2024
e8246a6
added settings.json export and download; path need adjustment to session
yyyxam Jan 5, 2024
27497e9
implemented session-path for settings.json creation (and download)
yyyxam Jan 6, 2024
80aa274
included tools in config export and import | only updating backend da…
yyyxam Jan 7, 2024
45498d8
messing around with application of imported config and sliders
yyyxam Jan 7, 2024
9b1ae8e
config-import now to front-end, which triggers update_config_store (o…
yyyxam Jan 19, 2024
12f7577
refactoring CSS properties
yyyxam Jan 20, 2024
1021f02
config import and export now include cam
yyyxam Jan 20, 2024
b3a1348
WIP: File-creation on Download-Request
yyyxam Jan 20, 2024
7117d25
Add docstring; Bug-hunting; Reformatting; Renaming
yyyxam Jan 26, 2024
dd51730
Merge remote-tracking branch 'origin/dev' into dev
yyyxam Jan 26, 2024
092556e
Refactored model_list.json and model_paths.json to one file; save_dos…
yyyxam Mar 9, 2024
4e58a53
Refactored reset_sliders to use match-case; tested individual slider_…
yyyxam Mar 20, 2024
1b59816
add to gitignore
yyyxam May 18, 2024
de591c0
merge-conf
yyyxam May 18, 2024
6f54e1d
Add comment for starting up gunicorn
yyyxam May 18, 2024
afd6f4c
edit comments
yyyxam May 19, 2024
b881794
clientside cb versuche
yyyxam May 19, 2024
2438641
implement dataframe caching
yyyxam May 19, 2024
8ee60b3
caching mit gunicorn
yyyxam May 19, 2024
b9dbe15
implement session-relation to caching
yyyxam May 20, 2024
4ca4c35
finish up caching by deleting old dcc.store references
yyyxam May 20, 2024
ecd845f
rol back clientside_callbacks.js for upcoming branch
yyyxam May 20, 2024
1538221
gitignore cache-directory content
yyyxam May 20, 2024
d0a24de
clean up temp-files, old docker tries, etc
yyyxam May 20, 2024
a12cfa6
re-add model file for MALA
yyyxam May 20, 2024
a9f2011
set cache-threshold for 20 concurrent users
yyyxam May 20, 2024
ea07f5b
Merge pull request #11 from mala-project/caching
yyyxam May 20, 2024
1900983
Merge remote-tracking branch 'origin/dev' into dev
yyyxam May 20, 2024
09323d3
add gunicorn as -; fix vulnerable dependencies
yyyxam May 20, 2024
d09cdfa
design adjustments (alignment, proper use of icons, color-schema, ...)
yyyxam May 20, 2024
e621a51
Update README.md
yyyxam May 20, 2024
de4152c
Update README.md
yyyxam May 20, 2024
37adb62
Update README.md
yyyxam May 24, 2024
b416e27
update ignore
yyyxam Jun 8, 2024
18ef537
Merge remote-tracking branch 'origin/dev' into dev
yyyxam Jun 8, 2024
aabd9d0
Delete .idea directory
yyyxam Jun 8, 2024
f39c01a
switch to graph_object 3d_scatter
yyyxam Jun 9, 2024
bfc4912
remove pycache
yyyxam Jun 11, 2024
feb5d08
remove .idea dir
yyyxam Jun 11, 2024
ba45f08
finish clean up of cache and idea files
yyyxam Jun 11, 2024
6e4d1dd
add new test_data for debug inference
yyyxam Jun 11, 2024
ea59877
make marker size absolute for main trace of scatter
yyyxam Jun 11, 2024
bb028fb
apply absolute size to atoms preview
yyyxam Jun 11, 2024
2e8cc73
remove caching; remove draggable slicing
yyyxam Jun 21, 2024
8802657
Move setups for docker
yyyxam Jun 21, 2024
9130747
back to old debug data; resize atoms
yyyxam Jun 21, 2024
91c0af0
clean up unused parameter session_id; add TODO for problem saving files
yyyxam Jun 23, 2024
5af89d2
reset standard particle_size in Main-plot; refactor ase.Atoms_to_df c…
yyyxam Jun 23, 2024
cd093ca
fix wrong download path
yyyxam Jun 23, 2024
d099371
fix wrong download path2
yyyxam Jun 23, 2024
bc37019
fix version in setup.py
yyyxam Jun 23, 2024
40b4589
fix branch in setup.py
yyyxam Jun 23, 2024
331a81e
remove old debug data; remove unused clientside_callbacks.js
yyyxam Jun 23, 2024
bcd69a9
refactor model_list to config file, including atom-limit and move it …
yyyxam Jun 23, 2024
56a2a32
open cfg in read mode
yyyxam Jun 23, 2024
26965f1
Update README.md
yyyxam Jun 23, 2024
de12a81
Dockerfile, .yaml, refactor req.txt
yyyxam Jun 23, 2024
2933f59
Merge remote-tracking branch 'origin/dev' into dev
yyyxam Jun 23, 2024
ea92dc3
Create LICENSE
yyyxam Jun 24, 2024
24ad72a
recolor and resize
yyyxam Jun 24, 2024
d6c7b70
restructure according to https://github.com/bradley-erickson/dash-app…
yyyxam Jun 24, 2024
2dd9901
fixing imports for previous packaging
yyyxam Jun 24, 2024
0802f5a
Merge remote-tracking branch 'origin/dev' into dev
yyyxam Jun 24, 2024
e1ee19a
add test input data
yyyxam Jul 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 20 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,21 @@
/lammps_input.tmp
/lammps_log.tmp
/upload
# don't ignore:
!requirements.txt

/src/session/
/src/lammps_bgrid_log.tmp
/src/lammps_input.tmp
/src/mala.pw.scf.in
/test/
/test_data/
/src/mala.pw.scf.in
**/__pycache__/
**/.idea/
/src/cache-directory/
*.tmp
*.iml
*.xml
*.txt
/malaWeb.egg-info
/.idea
*.pyc
/session/
1,628 changes: 0 additions & 1,628 deletions Be2_density.cube

This file was deleted.

25 changes: 25 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Use a slim Python 3 image as the base
FROM python:3.11-slim
LABEL authors="Maximilian Wenger"


# Create a working directory for the application
WORKDIR /app

# Copy the requirements.txt file
COPY requirements.txt .

# Install dependencies using pip
RUN pip install --no-cache-dir -r requirements.txt

# Copy the application code
COPY install .

# Install any additional system dependencies (if needed)
# RUN apt-get update && apt-get install -y some-package

# Expose the port used by Dash app (usually 8050)
EXPOSE 8050

# Use gunicorn as the application server (recommended for production)
CMD ["gunicorn", "--bind", "0.0.0.0:8050", "server:app"]
26 changes: 26 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
BSD 3-Clause License

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.

3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
79 changes: 67 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,72 @@
# malaWeb
Web app to visualize on-the-fly MALA predictions.

Note: So far, this only renders test-data for BE2. NOT the data that is uploaded.
There is only a file-format-check for ASE-readability happening. If successful, a hardcoded inference-script is run.
The atoms listed in the upload-popup are read from file though.
Web app to visualize on-the-fly [MALA](https://github.com/mala-project/mala) predictions.

# Installation:
## Description

- MALA needs to be installed together with Lammps
- When using Anaconda for MALA:
- conda install -c conda-forge dash
- conda install -c conda-forge dash-bootstrap-components
- pip install dash-uploader --pre
malaWeb is a Plotly Dash based web application used for the visualization of 3D volumetric data. By giving atom positions and cell information in ASE-accepted data formats, an inference can be run via MALA to make predictions on the volumetric data inside the given cell. This application is meant to be hosted for private access / in institutions and only in intranet. Security concerns have not been part of the development process so far and they definetly exist. \

# Usage:
- run app.py

Be aware that larger model predictions will take a lot of time. The default atom limit for predictions is 200.

## Installation

Running a MALA inference requires a working installation of MALA and its dependencies (torch, LAMMPS, QuantumEspresso, ..), as well as model data.\
See https://github.com/mala-project/mala/blob/develop/docs/source/install/installing\_mala.rst on how to install.\
Some model data is currently included in this malaWeb repository, stored in "models"-folder and listed in "model\_list.json".
After MALA is installed, just run the setup.py file to install malaWebs dependencies.

Options for installing malaWeb are:
* As recommended for MALA too, use a conda environment: run `conda env create --name <environment name> --file environment.yaml`.
* Install via pip: In the install directory, run `pip install -e .`
* Install and deploy with Docker: ...

Make sure the installed version of the dash uploader component is the [pre-release version 0.7.0](https://github.com/fohrloop/dash-uploader#-dash-uploader-070-pre-release-available) or newer, as this introduces new syntax.

Make sure the installed version of packaging is not higher than 21. For a version check, the dash uploader component uses an attribute "LegacyVersion" that has been deprecated in later versions. See this thread: https://github.com/pypa/packaging/issues/321

## Usage

After installing with conda or pip on host, run `gunicorn app:server -w 4 -b [host-ip]:8050`
Change 8050 to whatever port you want and exchange [host-ip] for the IP adress of the server.
Other devices on the same network can now access malaWeb via that IP:port.

If you plan to install and deploy with docker, in line X of the Dockerfile, change "0.0.0.0:8050" to you desired IP-adress and port as with the gunicorn command above.

\
In the File-Upload section, upload an ASE-readable file\
(See: https://wiki.fysik.dtu.dk/ase/ase/io/io.html -> Table -> Formats with either R or RW capabilities).\
The data read by ASE will be displayed in a popup window for running an Inference

<figure><img src="./src/assets/images/inference-popup.png" alt=""><figcaption><p>Inference popup</p></figcaption></figure>

* A dropdown lists all uploaded atom positions
* The given cell is rendered with given atom positions inside
* A model has to be chosen for running the MALA inference. Some models support varying temperatures, so make sure to edit if necessary, before running (calculation heavy) predictions
* Run MALA



<figure><img src="./src/assets/images/webapp-overview.png" alt=""><figcaption><p>Overview of malaWeb</p></figcaption></figure>

* (1) The interactive visualization will open up.
* (2) The Sidebar to the left contains:
* Upload section
* [new] Download button that saves MALA's results locally (needs work)
* The edit button for opening up the Inference pop up again, to enable running another inference (f.e. same data, different model)
* The reset button to delete the uploaded data and reset the visualization
* (3) The Settings panel on the right hand side gives options for:
* &#x20;Predefined camera angles
* Rendering options for the 'voxels' and their size, opacity and outlines as well as atoms and cell boundaries.
* [new] Import of config (settings, tools, camera angle) in JSON format
* [new] Download of config (settings, tools, camera angle) in JSON format
* (4) The Tools panel just below the visualization lets you render layers in all 3 axis, as well as filter voxels by their density value. Layer settings can quickly be en- and disabled by pressing the respective axis button and reset by pressing X to the sliders right.
* (5) The bottom of the page has a button for opening up information on different energies and a graph of the density of state.

## Todos
* Explore Client-Side plot updates for slicing
* Fix Download of MALA's data

Long Term:
* Create a MALA-API that grants access to a (powerful / optimized) MALA-Host, to reduce load on App-host and improve inference speed
* Import option for MALA data, so that malaWeb can also be used for just visualizing
Loading