I am doing some bioinformatics work in a shell terminal on a remote server, and supposedly I crashed our server with a massive file (which I could only assume was maybe a bomb).
I used the command below while attached to a screen.

zcat 05*1.p* 5_1.p* 6_1.p* 8_1.p* Ae*1.p* | gzip -c > Ae_Trimmed_1.paired.fq.gz & zcat 05*2.p* 5_2.p* 6_2.p* 8_2.p* Ae*2.p* | gzip -c > Ae_Trimmed_2.paired.fq.gz & zcat 1_1.p* 2_1.p* 3_1.p* 4_1.p* All1*1.p* | gzip -c > Ol_Trimmed_1.paired.fq.gz & zcat 1_2.p* 2_2.p* 3_2.p* 4_2.p* All1*2.p* | gzip -c > Ol_Trimmed_2.paired.fq.gz

The Ae_Trimmed_2.paired.fq.gz file grew over two weeks and after reaching a few terrabytes crashed our server.
I was trying to run four different commands linked by & but I obviously should have used &&
I read online for a bit so I understand the traditional functions of both now, but what does & do if used between multiple commands like above?

This post was edited by Badgerboys on Aug 2 2017 01:12pm

It starts them in parallel instead of in sequence.

zcat decompresses files, gzip compresses back, and all while heavy disk usage.

How does starting them in parallel cause the second file produced to continue growing?
I'm not very familiar with the concept.

Server most likely died due to disk IO and cpu usage, it should never ever crash because of full disk.

Because you ran it in parallel, there were 4 processes compressing / uncompressing and heavily reading/writing to disk at a time. It still shouldn't crash but then it's understandable, disk IO from my experience is #1 cause.

This post was edited by nuvo on Aug 2 2017 04:14pm

Ripp