I'm mainly just curious ... I'm not an admin, so I have no power to change the setup of the grids I use anyway ... are there any grid management software options out there that allow sensible piping and redirection in batch submit commands?
I'm a fairly heavy user of pipes, redirections, tees etc in my shell scripts on my own workstation ...
A lot of the files I work with are pretty large, so unless I'm doing something very repetitive, I would never really want to have them in an uncompressed state and I generally redirect stdin when working with them.
Most of the clusters I've used run on SGE, which happily swallows stdout and writes everything to files, which prevents me running things like the above. I think SLURM does a similar thing.
Has anyone used a system that actually allows a script like that to be submitted in array / batch commands? Or am I just hoping for something that doesn't exist?
I'm a fairly heavy user of pipes, redirections, tees etc in my shell scripts on my own workstation ...
Code:
prog1 <(zcat $foo.gz) <(zcat file1.gz) \
| prog2 \
| tee $foo.1.txt \
| prog3 \
> $foo.2.txtMost of the clusters I've used run on SGE, which happily swallows stdout and writes everything to files, which prevents me running things like the above. I think SLURM does a similar thing.
Has anyone used a system that actually allows a script like that to be submitted in array / batch commands? Or am I just hoping for something that doesn't exist?
Last edited: