v0.3.3

You may download the latest release with:

git clone -b v0.3.3 https://www.github.com/angelolab/ark-analysis.git

Finally, we need to upgrade the toffy version to 0.3.3

• Bump toffy to 0.3.3 @alex-l-kong (#479)
  @alex-l-kong

v0.3.2

You may download the latest release with:

git clone -b v0.2.3 https://github.com/angelolab/ark-analysis.git

Fixes an additional issue still remaining from v0.3.1, which mistakenly had mibi-bin-tools still pinned to 0.2.12.

🧰 Maintenance

• Fix mibi-bin-tools release to 0.2.14 @alex-l-kong (#478)
  @alex-l-kong

v0.3.1

You may download the latest release with:

git clone -b v0.2.3 https://github.com/angelolab/ark-analysis.git

This release serves as a springboard from v0.2.3, which was mistakenly tied in to tag v0.3.0 making it difficult to download on PyPI.

🧰 Maintenance

• Fix release issue inconsistency on PyPI @alex-l-kong (#477)
  @alex-l-kong

v0.2.3

You may download the latest release with:

git clone -b v0.2.3 https://github.com/angelolab/ark-analysis.git

What's new?

Noodle range extraction fix

Previously, users had run into issues with extremely dim Noodle signal. This was caused by an incorrect extraction range, which has been corrected by updating the range in the panel file of the cohort. With this correction, commercial_rosetta_matrix_v1.csv has again become the default matrix to use, as the _v2 matrix was applied over the incorrectly-extracted Noodle channel.

Backward compatibility for panels with incorrect Noodle ranges at the time of extraction has also been added.

Batch effects across TMAs

The batch effects notebook has been generalized to provide summary statistics across all TMAs. This can help isolated systematic issues that may occur across different TMA slides.

Autogain support

Ionpath has introduced the autogain feature, which reduces (but does not completely eliminate) the need for normalization. Additional visualization tools and support are implemented in this release to support this new functionality. Backward compatibility for non-autogain runs is also in place.

Temporary alpineer hotfixes

Due to changes in alpineer to support updates by @bryjcannon to support ezSegmenter and generic masks in ark-analysis, some breaking changes were introduced to toffy. While alpineer gets updated, temporary changes have been made to ensure compatibility.

Rosetta tiling fixes

In some unusual cases, Rosetta tiling will fail due to a divide-by-zero error in case of extremely sparse signal. A patch has been added for this case.

🚀 Features

• Bump commercial Rosetta matrix to v2 @alex-l-kong (#460)
• Adjust the batch effect QC notebook to average stats across all TMAs @camisowers (#458)
• Prevent pulse_height_fit from throwing dtype warning with floats @alex-l-kong (#456)
• Add support for MIBI runs acquired using autogain feature @alex-l-kong (#455)
• For autogain runs, add voltage plotting scheme @alex-l-kong (#453)

🐛 Bug Fixes

• Normalize by maximum pixel in case of Rosetta tiling for sparse signal @alex-l-kong (#472)
• Support legacy panel files defined before the Noodle range extraction fix @alex-l-kong (#470)
• Enforce exact_match=True when listing JSON file for get_estimated_time for MPH @alex-l-kong (#467)
• Fix Noodle extraction range in convert_panel() @camisowers (#464)
• Normalization notebook bug @camisowers (#461)

🧰 Maintenance

• Release v0.2.3 @alex-l-kong (#476)
• Bump mibi-bin-tools to 0.2.14 @alex-l-kong (#475)

📚️ Documentation

• Fix normalization documentation @camisowers (#454)

@alex-l-kong, @camisowers, and @srivarra

v0.2.2

The latest release of toffy features updated Rosetta Compensation, the deprecation of the tiling notebook, user experience improvements, along with documentation improvements and dependency housekeeping.

🚀 Features

• Check for all run merge conflicts @camisowers (#435)
• Partial FOVs clarification @camisowers (#419)
• Check for missing fovs after run is complete @camisowers (#410)
• Helper function for scaling images @camisowers (#415)

🐛 Bug Fixes

• Prevent mass column from adding during rosetta.combine_compensation_files @alex-l-kong (#444)

• Ensure low sensitivity normalization correction works with negative values too @alex-l-kong (#428)

• Use updated round 2 Rosetta folder path spec @alex-l-kong (#422)

• Space optimization for tiled Rosetta images @alex-l-kong (#421)

• Fix rosetta comparison image output @camisowers (#451)

• Add scaling to rosetta source channel images @camisowers (#440)

• Provide more robust round 1 -> round 2 copying @alex-l-kong (#426)

• Standardize indexing of over- and under-normalized channels in normalize_fov @alex-l-kong (#427)

🧰 Maintenance

• Release v0.2.2 @srivarra (#452)
• Updated the toffy environment file @srivarra (#434)
• Ensure safe conversion of panel types during modify_panel_ranges @alex-l-kong (#443)
• Update pulse height generation to be compatible with mibi-bin-tools==0.2.11 @alex-l-kong (#436)
• jupyter_contrib_nbextensions fix @srivarra (#429)

📚️ Documentation

• Indicate incompatible normalization notebook with MIBIcontrol 1.8 @alex-l-kong (#418)
• MIT License @srivarra (#448)
• Add pre-commit to standardize toffy documentation @camisowers (#437)
• Cite TSAI in tiling notebooks, deprecating them @alex-l-kong (#420)

@alex-l-kong, @camisowers and @srivarra

v0.2.1

The latest release of toffy introduces major updates to the watcher pipeline, removing duplicate data generation, re-extracting incomplete .bin files, and enabling startup without a run folder. It introduces mass proficient image extraction to address storage concerns from large .bin files, round 2 Rosetta for additional denoising, standardized curves and outlier capping for post-Rosetta normalization, and post-reorganization QC metrics to verify across TMAs. Additional features include improved tiled image stitching for TMA + tissue samples, FOV resolution QC checking, and standardized output folders on the D drive for MIBI computers.

🚀 Features

• Check for partially generated fovs @camisowers (#416)
• Remove hanging extracted_imgs_dir reference in Rosetta notebook @alex-l-kong (#413)
• QC Metrics - Save Longitudinal Transformed Metrics @srivarra (#408)
• Ensure watcher can wait on run folder to appear @alex-l-kong (#401)
• QC for TMAs and Control Samples @srivarra (#358)
• FOV resolution check @camisowers (#391)
• Adjust rosetta processing paths @camisowers (#385)
• Tma and tiled run stitching @camisowers (#382)
• Mass proficient extraction for NAS storage @alex-l-kong (#342)
• Add Rosetta V2 processing functionality @alex-l-kong (#319)
• Ensure runs below fovs_per_num threshold get skipped during Rosetta t… @alex-l-kong (#406)
• Skip watcher callbacks for FOVs that already have data generated @alex-l-kong (#394)
• Ensure incompletely extracted bin files get re-extracted @alex-l-kong (#392)
• Turn off combined.csv file warnings for intermediate callbacks in watcher @alex-l-kong (#393)
• Turn on y ticks for QC plots @alex-l-kong (#386)
• Norm workflow @alex-l-kong (#361)
• Cap normalization at 10X upper bound, and 1X lower bound @alex-l-kong (#352)
• Add functionality to catch file renaming to watcher @alex-l-kong (#359)
• Remove warning for .def.hist and .pro.hist files @alex-l-kong (#381)
• Allow for image specified for flatfield correction to be empty @cliu72 (#383)

🐛 Bug Fixes

• Fix rescale_raw_images error @camisowers (#398)
• Ensure pulse heights file is named with _ not - @alex-l-kong (#412)
• Make sure watcher logging actually writes to file on CACs @alex-l-kong (#390)
• Add a scale factor for image_stitching.stitch_images for Photoshop compatibility @alex-l-kong (#374)

🧰 Maintenance

• Update environment.yml @srivarra (#372)
  @alex-l-kong, @camisowers, @cliu72 and @srivarra

v0.2.0

The latest release of toffy includes some bug fixes to median pulse heights, normalization for all-zero pulse height channels and intermediate callback plotting per FOV, as well as updating multiple tile stitching to be more flexible. In addition a few small bugs were fixed such as missing imports, and updating documentation for installing pre-commit. Our GitHub Actions pipeline has also received a light cleanup.

🚀 Features

• GitHub Actions - CI Updates @srivarra (#370)
• Multiple tile stitching @camisowers (#360)

🐛 Bug Fixes

• Refresh intermediate callback plotting on each new FOV @alex-l-kong (#363)
• Fix median pulse height chart drawn by visualize_mph @alex-l-kong (#356)
• Added conda-forge channel to environment.yml @srivarra (#354)
• Ensure normalization doesn't happen for all-zero pulse height channels @alex-l-kong (#326)
• Update axes_size to axes_font_size in 3c notebook @alex-l-kong (#343)
• Missing import in extraction notebook @camisowers (#334)

🧰 Maintenance

• Next Release - v0.1.3 @srivarra (#371)

📚️ Documentation

• Fix brew install command for pre-commit @alex-l-kong (#341)
• Update README.md @ngreenwald (#333)
  @alex-l-kong, @camisowers, @ngreenwald and @srivarra

v0.1.2

Adjusts toffy to use the new alpineer repo containing shared utility functions between toffy and ark-analysis. Also adds in a few new features such as tiled image stitching and real time plotting for the watcher notebook. Switches from Travis-CI to GitHub Actions.

🚀 Features

• GitHub Actions @srivarra (#328)
• Tiled stitching for toffy @camisowers (#285)
• Flexible Requirements @srivarra (#330)
• Add ability to override run callbacks to process per FOV @alex-l-kong (#313)
• Update Rosetta test directory setup @alex-l-kong (#300)

🐛 Bug Fixes

• Skip empty json files in voltage check @camisowers (#290)
• Don't display warning if the watcher discovers a temp file @alex-l-kong (#329)
• Remove MIBItracker tests @alex-l-kong (#323)
• Make sure Rosetta converts images to float prior to running @alex-l-kong (#293)

🧰 Maintenance

• Conda Environment - scikit-learn & seaborn @srivarra (#297)
• Substituted Ark imports with TMI in the Jupyter Notebooks @srivarra (#295)
• Adding TMI @srivarra (#286)
• add tmi to environment.yml @camisowers (#284)
• Image Utils in TMI @srivarra (#282)
  @alex-l-kong, @camisowers and @srivarra

v0.1.1

Introduces small bug fixes for loading in panels, extraction crash logic improvements with json files and lossless compression for tiffs on by default.

🚀 Features

• Compressed Tiff Files @srivarra (#280)

🐛 Bug Fixes

• Make sure loading in convert_panel happens in utf-8 @alex-l-kong (#271)
• Fix extraction logic for run with a single fov crash @camisowers (#269)

🧰 Maintenance

• Next Release - v0.1.1 @srivarra (#281)

📚️ Documentation

• Add link in notebooks for the ark tiled Image Stitching script @camisowers (#278)
• Toffy Setup Video Fix @srivarra (#267)
  @alex-l-kong, @camisowers and @srivarra

v0.1.0

First release of the beta version of toffy. This includes new MIBI image generation and processing tools, organized into a multistep pipeline for users to follow. Also contains adjustments made in preparation for and following the MIBI Workshop.

🚀 Features

• Notebook 1 normalization checks @camisowers (#250)
• Add _pulse_height.csv file generation to watcher @camisowers (#262)
• Custom error for Rosetta file copying @camisowers (#258)
• Json file encoding @camisowers (#243)
• Stitch images function without run file @camisowers (#237)
• Toffy notebook fixes @camisowers (#230)
• Create release-drafter.yml @srivarra (#223)
• Rosetta stitching fix @camisowers (#227)
• Add interactive tiled region overlay @alex-l-kong (#86)
• Add temporary scripts directory to track notebook updates @alex-l-kong (#220)
• Rosetta notebook testing updates @camisowers (#212)
• Optimize Watcher Tests @ackagel (#219)
• Remove moly_path requirement for remap_to_reorder_fovs @alex-l-kong (#215)
• Update run metrics saving behavior @camisowers (#213)
• Add checks for out-of-bounds coordinates during tiling process @alex-l-kong (#209)
• Panel conversion implementation @camisowers (#202)
• Added .DS_Store in the .gitignore @srivarra (#206)
• qc metrics violin plots on normal tissues @mheskett (#153)
• Ensures co-registration path is absolute @alex-l-kong (#196)
• Image stitching in watcher @camisowers (#152)
• Adjust min TMA size to 2x2 @alex-l-kong (#194)
• Simplify QC metric visualization procedure @alex-l-kong (#174)
• Output float32 images for normalization notebook @camisowers (#155)
• run fovs in numerical order @camisowers (#154)
• Check voltage @camisowers (#148)
• Split run file @camisowers (#145)
• Update QC notebook with real data @camisowers (#147)
• Extract bin notebook @camisowers (#130)
• Mph plots @camisowers (#101)
• Mph histograms @camisowers (#126)
• Implement json helper funcs @mheskett (#142)
• Json helper functions @mheskett (#138)
• Small tweaks @ngreenwald (#137)
• Format data for analysis @ngreenwald (#131)
• added moly_path option @srivarra (#114)
• Mph norm workflow @ngreenwald (#125)
• find moly FOVs based on JSON file @ngreenwald (#124)
• Rosetta simplification @ngreenwald (#107)
• implements watcher chaining architecture @ackagel (#99)
• Rosetta modifications @ngreenwald (#92)
• Rename warning @camisowers (#103)
• Update repo setup for watcher @ngreenwald (#93)
• Rename fov folders @camisowers (#85)
• Streak Detection @srivarra (#71)
• adds fov_watcher template @ackagel (#53)
• Don't check contrast for skimage save function. @srivarra (#58)
• Add ROI name prefix to tiled region FOV names @alex-l-kong (#66)
• Rosetta workflow @ngreenwald (#43)
• Write co-registration settings to JSON param file @alex-l-kong (#60)
• Add watcher compatible QC notebook @alex-l-kong (#50)
• Add notebook to change co-registration parameters in settings.py @alex-l-kong (#45)
• Add tiled regions notebook to toffy @alex-l-kong (#41)
• Normalize images by MPH @ngreenwald (#35)
• Change x to columns and y to rows in TMA script @alex-l-kong (#39)
• Add TMA FOV validation functions for user sanity checking @alex-l-kong (#30)
• Add TMA tiling utils into creed @alex-l-kong (#13)
• Support non-rectangular TMAs @ngreenwald (#19)
• Automatic checking of hashes for transferred files @ngreenwald (#21)
• Add automatic detector sweep ID @ngreenwald (#17)
• Add Rosetta matrix functionality @ngreenwald (#27)
• Add coveralls support @alex-l-kong (#14)
• Add Travis and pytest support @alex-l-kong (#15)
• Add notebooks @ackagel (#3)

🐛 Bug Fixes

• get_max_img_size works with subdir @camisowers (#264)
• Rosetta mass 39 naming conflict @camisowers (#252)
• Panel conversion column typo @camisowers (#247)
• Ignore stitched_images folder in rosetta processing @camisowers (#239)
• Ensure path to rosetta_matrices is added in the setup notebook @alex-l-kong (#236)
• Catches tempdir cleanup error @camisowers (#225)
• Make watcher skip moly points @ackagel (#117)
• Remove endline in column names from correctly formatted panel @camisowers (#211)
• Fix pulse_height_dir path in normalize notebook @alex-l-kong (#210)
• Watcher hidden files @camisowers (#193)
• Added the location of the Optical Image File in the Notebook @srivarra (#182)
• Empty json check @camisowers (#187)
• Make sure mapping_path load call gets assigned to mapping prior to relabeling TMA FOV names @alex-l-kong (#186)
• Adjusted Streak Detection mask generation parameters @srivarra (#119)
• Fix import error @camisowers (#96)
• Add scheme for renaming TMA manual FOV duplicates @alex-l-kong (#84)
• Ensure JSON data is read in with encoding UTF-8 @alex-l-kong (#36)
• Change notebook import from creed to toffy @alex-l-kong (#33)
• Update repo for toffy @ngreenwald (#32)

🧰 Maintenance

• Adjusted requirements to work with ark-analysis on Python 3.7 @srivarra (#132)
• Pin ark requirement @ngreenwald (#121)
• Updating the License @srivarra (#47)

📚️ Documentation

• Add workshop links to readme @camisowers (#260)
• Update flowchart in readme @camisowers (#248)
• Update README.md with link to discussions @ngreenwald (#229)
• Reorg notebook cleaning @camisowers (#222)
• Add documentation for TMA notebook @alex-l-kong (#207)
• Update readme with directory structure @camisowers (#190)
• Added Step 3e to the pipeline flowchart @srivarra (#204)
• Notebook 4a - Documentation Improvements @srivarra (#177)
• Toffy Pipeline Flowchart @srivarra ([#200](https://github....