Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add async support to neomodel #783

Merged
merged 59 commits into from
Apr 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
b6c9d48
Propagating async from AsyncDriver ; refactoring Database
Nov 23, 2023
6bcb97a
Fix session scope fixtures ; and a first test
Nov 23, 2023
3d30cd1
First test working
Dec 8, 2023
0d92dab
Add auto unasync with pre commit
Dec 11, 2023
8039635
Remove async suffix for methods in async classes
Dec 11, 2023
d8f5855
Merge remote-tracking branch 'origin/master' into task/async
Dec 12, 2023
26a63c4
Move Async in class name to prefix
Dec 12, 2023
9890407
Fix token replacements
Dec 15, 2023
12b0601
Fix sync version
Dec 15, 2023
60581f8
Some fixes to make-unasync
Dec 19, 2023
ebe0cb7
Run mak-unasync with python 3.11
Dec 19, 2023
34d2847
Fix scripts - sync code
Dec 19, 2023
eff0928
Fix unasync script for Python 3.12
robsdedude Dec 19, 2023
3b574e5
Migrate more to async
Dec 20, 2023
0c84352
Make Relationships async
Dec 20, 2023
a95ea04
Rename _async to sync_ ; Update autodoc
Dec 22, 2023
c288324
Fix async_ replacements
Dec 22, 2023
cd72533
Some more fixes and test
Dec 29, 2023
c9b877e
Fix donder methods
Dec 29, 2023
d4ea579
Extract PropertyManager for async reasons
Dec 29, 2023
8f90b4b
Fix cardinality and test
Dec 29, 2023
7559516
More tests
Dec 29, 2023
9fd375a
More tests
Jan 2, 2024
b17f868
Fix non-preserved order
Jan 2, 2024
db05e4f
More tests
Jan 2, 2024
8bba824
More tests
Jan 2, 2024
bd7c298
More tests and fixes
Jan 3, 2024
45120ea
Rename sync test folder
Jan 3, 2024
f3a6429
More tests and isort tests
Jan 3, 2024
c55ad91
Fix import error test contrib
Jan 3, 2024
681f4f6
Migrate test_contrib to async
Jan 3, 2024
a38e7ec
Fix make-unasync for test_contrib
Jan 3, 2024
fa6b38f
Update pre-commit-config
Jan 3, 2024
6ac8094
Move test_contrib
Jan 3, 2024
8787a51
Remove commented out code
Jan 4, 2024
3d3308a
Merge remote-tracking branch 'origin/master' into task/async
Jan 4, 2024
94b9170
Update doc for async
Jan 4, 2024
d09e17e
Fix await MyLabel.nodes
Mar 4, 2024
7025cdb
Update tests
Mar 4, 2024
65facf2
Fix transaction decorator
Mar 11, 2024
95bdd1a
Fix return type hint
Mar 11, 2024
f428e72
Fix skipif
Mar 11, 2024
5b8825c
Propagate async database_version ; fix __await__ dunders
Mar 16, 2024
bc88479
Revert pre-commit
Mar 16, 2024
aba3a02
Fix tests
Mar 16, 2024
a83ae86
Move singleton import into root
Mar 16, 2024
a62c054
Branch tests async/sync
Mar 16, 2024
dd5c373
Update doc
Mar 16, 2024
2d7028b
Remove flay type hint
Mar 16, 2024
df0efa0
FIx element id parsing
Mar 18, 2024
3219d08
Fix pre-commit
Mar 19, 2024
bb03faa
Apply isort in unasync
Mar 25, 2024
0dc41b3
Fix transactions ; Improve make-unasync
Mar 25, 2024
da1c0d8
Fix check coroutine
Mar 25, 2024
ff058d9
Fix test order
Mar 26, 2024
d4b48ba
Fix sync conftest
Mar 28, 2024
a229418
Fix numpy test
Mar 28, 2024
8bd7aa0
Fix some code smells
Mar 28, 2024
ce3a1cf
Update README with performance tests
Apr 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ development.env
.ropeproject
\#*\#
.eggs
bin
lib
.vscode
pyvenv.cfg
Expand Down
25 changes: 13 additions & 12 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,17 +1,18 @@
repos:
- repo: https://github.com/psf/black
rev: 22.3.0
hooks:
- id: black
- repo: https://github.com/PyCQA/isort
rev: 5.11.5
hooks:
- id: isort
# - repo: local
# hooks:
# - id: pylint
# name: pylint
# entry: pylint neomodel/
# language: system
# always_run: true
# pass_filenames: false
args: ["--profile", "black"]
- repo: https://github.com/psf/black
rev: 23.3.0
hooks:
- id: black
- repo: local
hooks:
- id: unasync
name: unasync
entry: bin/make-unasync
language: python
files: "^(neomodel/async_|test/async_)/.*"
additional_dependencies: [unasync, isort, black]
52 changes: 50 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ GitHub repo found at <https://github.com/neo4j-contrib/neomodel/>.

# Documentation

(Needs an update, but) Available on
Available on
[readthedocs](http://neomodel.readthedocs.org).

# Upcoming breaking changes notice - \>=5.3
Expand All @@ -47,7 +47,7 @@ support for Python 3.12.

Another source of upcoming breaking changes is the addition async support to
neomodel. No date is set yet, but the work has progressed a lot in the past weeks ;
and it will be part of a major release (potentially 6.0 to avoid misunderstandings).
and it will be part of a major release.
You can see the progress in [this branch](https://github.com/neo4j-contrib/neomodel/tree/task/async).

Finally, we are looking at refactoring some standalone methods into the
Expand All @@ -67,6 +67,15 @@ To install from github:

$ pip install git+git://github.com/neo4j-contrib/neomodel.git@HEAD#egg=neomodel-dev

# Performance comparison

You can find some performance tests made using Locust [in this repo](https://github.com/mariusconjeaud/neomodel-locust).

Two learnings from this :

* The wrapping of the driver made by neomodel is very thin performance-wise : it does not add a lot of overhead ;
* When used in a concurrent fashion, async neomodel is faster than concurrent sync neomodel, and a lot of faster than serial queries.

# Contributing

Ideas, bugs, tests and pull requests always welcome. Please use
Expand Down Expand Up @@ -112,3 +121,42 @@ against all supported Python interpreters and neo4j versions: :

# in the project's root folder:
$ sh ./tests-with-docker-compose.sh

## Developing with async

### Transpiling async -> sync

We use [this great library](https://github.com/python-trio/unasync) to automatically transpile async code into its sync version.

In other words, when contributing to neomodel, only update the `async` code in `neomodel/async_`, then run : :

bin/make-unasync
isort .
black .

Note that you can also use the pre-commit hooks for this.

### Specific async/sync code
This transpiling script mainly does two things :

- It removes the await keywords, and the Async prefixes in class names
- It does some specific replacements, like `adb`->`db`, `mark_async_test`->`mark_sync_test`

It might be that your code should only be run for `async`, or `sync` ; or you want different stubs to be run for `async` vs `sync`.
You can use the following utility function for this - taken from the official [Neo4j python driver code](https://github.com/neo4j/neo4j-python-driver) :

# neomodel/async_/core.py
from neomodel._async_compat.util import AsyncUtil

# AsyncUtil.is_async_code is always True
if AsyncUtil.is_async_code:
# Specific async code
# This one gets run when in async mode
assert await Coffee.nodes.check_contains(2)
else:
# Specific sync code
# This one gest run when in sync mode
assert 2 in Coffee.nodes

You can check [test_match_api](test/async_/test_match_api.py) for some good examples, and how it's transpiled into sync.

Loading
Loading