Run too longtime

[ucsfpan]

Hi,
Thanks for the great tools. I am running Juicer 2.0 on 110–112 Gb gzipped FASTQ files from Arima using SLURM.
After ~12 days the job is still in the queue, but the lastest outputs are merged30.txt (26 Gb),
merged_dedup.bam (248 Gb), and the log shows
Exception in thread "pool-2-thread-28" java.lang.OutOfMemoryError: Java heap space

Could you advise how to speed-up or which internal step is blocking?
SLURM header and java/sort settings are attached.
No problems were observed on the small test set (hic003.fastq.gz).

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=30
#SBATCH --mem=300G
#SBATCH --time=300:00:00
#SBATCH --array=0
#SBATCH --output=%x-%A-%a.out

export _JAVA_OPTIONS="-Xmx32g -Xms32g"
export SORT_MEM_GB=6

scripts/juicer.sh
-d /data1/juicer/M100511
-D /data1/juicer/M100511
-y restriction_sites/hg38_Arima.txt
-z references/hg38.fa
-p restriction_sites/hg38.chrom.sizes
-s Arima
-g hg38
-t 30
-T 30
2> error.log hi

[moshe-olshansky]

I suspect that it fails in the 'pre' step (hic file creation). The easiest way to continue may be to edit the juicer.sh script and change the java memory option, setting it as high as your hardware allows (please note that this command runs on a single node). To set the maximum RAM available to you java program to, say, 300G do
java -Xmx300G juicer_tools.jar pre ...
Make sure that there is no space between -Xmx and 300G.

[ucsfpan]

Thanks so much for your guidance. I followed your suggestion and changed the command to
java -Xmx300G juicer_tools.jar pre …
Now the .hic file can be produced, but the pipeline still hangs at the step for 10days:
java -Djava.awt.headless=true -jar juicer_tools.jar addNorm --threads 30 aligned/inter_30.hic
The file opens in Juicebox, yet the default scale is not applied.
The log shows the following error:
java.util.zip.DataFormatException: incorrect header check
at java.util.zip.Inflater.inflateBytes(Native Method)
at java.util.zip.Inflater.inflate(Inflater.java:259)
at org.broad.igv.util.CompressionUtils.decompress(CompressionUtils.java:87)
at org.broad.igv.util.CompressionUtils.decompress(CompressionUtils.java:56)
at juicebox.data.DatasetReaderV2.decompress(DatasetReaderV2.java:1032)
at juicebox.data.DatasetReaderV2.readBlock(DatasetReaderV2.java:980)
at juicebox.data.DatasetReaderV2.readNormalizedBlock(DatasetReaderV2.java:922)
at juicebox.data.iterator.ContactRecordIterator.hasNext(ContactRecordIterator.java:85)
at juicebox.data.iterator.CoupledIteratorAndOffset.hasNext(CoupledIteratorAndOffset.java:44)
at juicebox.data.iterator.BigContactRecordList.populateListOfListsFromSingleIterator(BigContactRecordList.java:41)
at juicebox.data.iterator.ListOfListGenerator.lambda$populateListOfLists$0(ListOfListGenerator.java:91)
at juicebox.tools.dev.ParallelizedJuicerTools$1.run(ParallelizedJuicerTools.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Could you please advise how to proceed? Thank you very much!

[moshe-olshansky]

Hi,
Which version of juicer_tools are you using and what is the highest resolution you need?
If, for example, you don't need a resolution higher than 500bp, you can do the following:
java -Xmx300g -jar juicer_tools.jar pre -n -r 2500000,1000000,500000,250000,100000,50000,25000,10000,5000,2000,1000,500 ...
The -n flag yells pre to skip normalisation, and -r flag provides the list of resolutions. Otherwise it can build the map with a resolution of up to 1bp and then attempt to normalise it.
After this is done (which should take reasonable time) you can add normalisation:
java -Xmx100G -jar juicer_tools.jar addNorm -k VC,VC_SQRT,SCALE your,hic
You can also add the -d flag if you are only interested in intrachromosomal contacts.

Good luck!

[ucsfpan]

Hi!
I’m using Juicer v2.0. My issue is very likely caused by these reasons. Thanks for the tips, I’ll give them a try.