-
Notifications
You must be signed in to change notification settings - Fork 302
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Spatially chunked and sharding compatible annotation writer #522
Closed
Closed
Changes from 32 commits
Commits
Show all changes
49 commits
Select commit
Hold shift + click to select a range
40d27b9
initial sharding compatible version using cloudvolume
fcollman aa0b558
make cloud volume import optional
fcollman 04e623b
changed to write via tensorstore
fcollman b058f78
formatting fixes
fcollman 331b523
spatial sharding WIP
fcollman 196f8b5
fix big endian encoding of spatial index and description
fcollman 074b6e2
formatting fixes
fcollman 7862de0
change shard spec to NamedTuple
fcollman 4e6ebb8
make tensorstore import optional
fcollman 29e9f75
fixing typing errors
fcollman 6efd7bc
fixing type hinting
fcollman 8c9169a
fixing lint issues
fcollman 949f22b
fix more linting
fcollman fd77f40
fixing import linting
fcollman 3bc1a64
fix: numpy.cast removal
fcollman dc2fd57
another import block fix
fcollman cd1e0dd
ruff formatting
fcollman 5ce8665
feat: dynamic lower bounds
fcollman 8f08c24
removing annotation reader
fcollman f147932
remove unused max annotations
fcollman e6211c0
make chunk_size default dynamic to size of the coordinate system
fcollman 38eb529
fixing mypy with cast
fcollman 073573b
mypy fix: value error print formatting
fcollman aece02d
fix doc string
fcollman d8aa858
fix chunk_size typing
fcollman 34ac02a
renaming chunk size
fcollman b15286b
add better handling of multiple point annotations
fcollman d073bea
adding ellipsoid
fcollman 75ba944
fix ellipsoid logic
fcollman a2c1932
fixing typing
fcollman ed4761a
ruff formatting
fcollman bbabf64
fix: relationship key encoded incorrectly
fcollman 524cca2
fixing dtypes of chunk size
fcollman 5f80c3c
fix: missing related IDs in by_id index
fcollman 987659e
fix: typo
fcollman ca9f066
bugfix: byid endian encoding fix
fcollman 8119007
fixing np.asarray case
fcollman fcd47a6
fix single layer chunks
fcollman 667d203
fixing chunk size for single planes
fcollman cf5579e
fix num_chunks
fcollman 20c40e6
fixing spatial indices for points
fcollman e6398e8
fixed sharded spatial index writing
fcollman afbc23d
generlizing spatial bins with rtree
fcollman a8c7a0f
improved generalization of upper and lower bound
fcollman 2a5eeea
remove comments, add sharded option, fix generator
fcollman 1d502ee
adding rtree dependancy
fcollman a781732
switch to sharded writing as default
fcollman 066b801
removing chunks with no items
fcollman 109726a
Merge branch 'master' into annotationwriter_improvements
fcollman File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm running into a snag here.
TypeError: No loop matching the specified signature and casting was found for ufunc ceil
I moved the dtype to the asarray and it seemed to have worked.
downsampled_shape = np.asarray( np.ceil(self.shape / downsample_factor), dtype=np.int64 )
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks fixed!