Help with troubleshooting qiime dada2 denoise-paired error. [NAs produced by integer overflow...is there a fix?] #2076
Comments
|
I got the same error after running it through the dada2 pipeline. However, this pipeline shows me which sample was being processed when the error occurred, and it's the largest sample, by far (1.83GB for R2 alone). Is there some kind of file size limitation? |
|
We are starting to run into some integer overflow issues with really large samples, which is what you are experiencing. We've fixed overflows in a couple places in this commit: d8bb758 Which version of dada2 in R were you using? If you weren't using the latest devel version (the latest version of the "master" branch on Github) could you give that a try and see if it can handle the problem sample? |
I'm currently running it again, but this time I split that large sample half (now two separate files for R1 and R2) to see if that may fix the problem, but I will also try the latest version of the master branch on github with the original sample too! I'll post an update after. Thanks for helping! |
|
I upgraded dada2 to version 1.35.6 and still get the same error at the same sample (see below). I'm currently running the split sample in qiime. I'll let you know if that works.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I encountered the following error during the dada2 denoise-paired step after removing primers using cutadapt (see below for the command and error). The amplicon has an average length around 183 bp. I'm using qiime2-amplicon-2024.5 on a mac M3 max computer. I've run this exact code for many other libraries, all prepped exactly the same using the same assay, but have never encountered this error.
I'm currently trying to run it through the dada2 pipeline (not within qiime) to see if I still get the same error. I'll follow up once it finishes. In the meantime, I'm hoping someone here may have some insights. Thanks!
(qiime2-amplicon-2024.5) rdceradk@CERADK-MN-BB888 Trinity_Orchid_inverts_09Dec2024 % qiime dada2 denoise-paired
--i-demultiplexed-seqs demux-paired-end-trimmed.qza
--p-trunc-len-f 125
--p-trunc-len-r 125
--p-trim-left-f 0
--p-trim-left-r 0
--p-n-threads 15
--p-n-reads-learn 1000000
--o-representative-sequences rep-seqs-dada.qza
--o-table table-dada.qza
--o-denoising-stats stats-dada.qza
--verbose
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada.R --input_directory /var/folders/tf/9kbmk08x4y19x3rhtkqd_cjm0000gr/T/tmputofb3vt/forward --input_directory_reverse /var/folders/tf/9kbmk08x4y19x3rhtkqd_cjm0000gr/T/tmputofb3vt/reverse --output_path /var/folders/tf/9kbmk08x4y19x3rhtkqd_cjm0000gr/T/tmputofb3vt/output.tsv.biom --output_track /var/folders/tf/9kbmk08x4y19x3rhtkqd_cjm0000gr/T/tmputofb3vt/track.tsv --filtered_directory /var/folders/tf/9kbmk08x4y19x3rhtkqd_cjm0000gr/T/tmputofb3vt/filt_f --filtered_directory_reverse /var/folders/tf/9kbmk08x4y19x3rhtkqd_cjm0000gr/T/tmputofb3vt/filt_r --truncation_length 125 --truncation_length_reverse 125 --trim_left 0 --trim_left_reverse 0 --max_expected_errors 2.0 --max_expected_errors_reverse 2.0 --truncation_quality_score 2 --min_overlap 12 --pooling_method independent --chimera_method consensus --min_parental_fold 1.0 --allow_one_off False --num_threads 15 --learn_min_reads 1000000
R version 4.3.3 (2024-02-29)
Loading required package: Rcpp
DADA2: 1.30.0 / Rcpp: 1.0.12 / RcppParallel: 5.1.6
2) Filtering The filter removed all reads: /var/folders/tf/9kbmk08x4y19x3rhtkqd_cjm0000gr/T/tmputofb3vt/filt_f/4March2024-76_L001_R1_001.fastq.gz and /var/folders/tf/9kbmk08x4y19x3rhtkqd_cjm0000gr/T/tmputofb3vt/filt_r/4March2024-76_L001_R2_001.fastq.gz not written.
Some input samples had no reads pass the filter.
........................................................................................................................................................................................................................x............................................................
3) Learning Error Rates
312203250 total bases in 2497626 reads from 3 samples will be used for learning the error rates.
312203250 total bases in 2497626 reads from 3 samples will be used for learning the error rates.
3) Denoise samples ................................................................Error in dada(drpF, err = err, multithread = multithread, verbose = FALSE) :
NAs in derep$quals matrix. Check that all input sequences had valid associated qualities assigned.
In addition: Warning message:
In derepQuals[sqnms, ] + out$cum_quals[sqnms, ] :
NAs produced by integer overflow
2: stop("NAs in derep$quals matrix. Check that all input sequences had valid associated qualities assigned.")
1: dada(drpF, err = err, multithread = multithread, verbose = FALSE)
Traceback (most recent call last):
File "/opt/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/q2_dada2/_denoise.py", line 350, in denoise_paired
run_commands([cmd])
File "/opt/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/q2_dada2/_denoise.py", line 37, in run_commands
subprocess.run(cmd, check=True)
File "/opt/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['run_dada.R', '--input_directory', '/var/folders/tf/9kbmk08x4y19x3rhtkqd_cjm0000gr/T/tmputofb3vt/forward', '--input_directory_reverse', '/var/folders/tf/9kbmk08x4y19x3rhtkqd_cjm0000gr/T/tmputofb3vt/reverse', '--output_path', '/var/folders/tf/9kbmk08x4y19x3rhtkqd_cjm0000gr/T/tmputofb3vt/output.tsv.biom', '--output_track', '/var/folders/tf/9kbmk08x4y19x3rhtkqd_cjm0000gr/T/tmputofb3vt/track.tsv', '--filtered_directory', '/var/folders/tf/9kbmk08x4y19x3rhtkqd_cjm0000gr/T/tmputofb3vt/filt_f', '--filtered_directory_reverse', '/var/folders/tf/9kbmk08x4y19x3rhtkqd_cjm0000gr/T/tmputofb3vt/filt_r', '--truncation_length', '125', '--truncation_length_reverse', '125', '--trim_left', '0', '--trim_left_reverse', '0', '--max_expected_errors', '2.0', '--max_expected_errors_reverse', '2.0', '--truncation_quality_score', '2', '--min_overlap', '12', '--pooling_method', 'independent', '--chimera_method', 'consensus', '--min_parental_fold', '1.0', '--allow_one_off', 'False', '--num_threads', '15', '--learn_min_reads', '1000000']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/q2cli/commands.py", line 520, in call
results = self._execute_action(
File "/opt/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/q2cli/commands.py", line 581, in _execute_action
results = action(**arguments)
File "", line 2, in denoise_paired
File "/opt/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/qiime2/sdk/action.py", line 342, in bound_callable
outputs = self.callable_executor(
File "/opt/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/qiime2/sdk/action.py", line 576, in callable_executor
output_views = self._callable(**view_args)
File "/opt/miniconda3/envs/qiime2-amplicon-2024.5/lib/python3.9/site-packages/q2_dada2/_denoise.py", line 363, in denoise_paired
raise Exception("An error was encountered while running DADA2"
Exception: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.
Plugin error from dada2:
An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.
See above for debug info.
The text was updated successfully, but these errors were encountered: