nextflow: avoid work directory being generated despite output folder

I've following code that generates a out.txt in path I decide: results_new/check
The script runs fine, that is out.txt file is generated successfully, however, another out.txt is generated at:

/Users/name/Documents/user/nextflow_scripts/test/work/55/5d31c87911f64f9060cf4560ef381c/out.txt

params.hg38genome ="/Users/sariys01/Downloads/NM.fasta"
params.outdir = './results_new/'

process create_file {

	output:
  publishDir "${params.outdir}/check", mode: 'copy'
    path("out.txt"), emit: json

	script:
	"""
	echo "hello\n"
	touch out.txt
echo "$reads\n"
	"""
}

workflow {
create_file(params.hg38genome).view()
}

How do I avoid this work folder being generated?

You can't/shouldn't avoid this out.txt from being generated.

This comes from the principles of Nextflow. Roughly, when a process is started, a working directory is created. All the files listed in the input: of that process are linked into that directory, then the code within the process is run. So, from the point of view of the code being run, everything appears to happen in a perfectly normal directory full of files. The code being run can create some amount of output, in this directory it's being run in (i.e. the working directory). Then, when the process is finishing, the files listed in the output: are exported as appropriate. In your case, that means making a copy of out.txt in your publishDir.

So, by construction, the file out.txt can only be exported to the publishDir if it has been generated in the working directory.

Now, the question is why you would want to avoid it being generated there? Typically, it's something that happens in the background, you shouldn't need to even think about it in daily Nextflow usage. If it's storage concerns, note that you have two useful options.

First, the working directory itself is created at the path defined in the environment variable NXF_WORK. You can change this environment variable if you want to store it somewhere else.

Second, you can easily delete all these "temporary" files after the pipeline has run, using:

nextflow clean

But you don't want to do that systematically, these files can be useful! Indeed, before a process is run, it'll first check if the corresponding results already exist. And if so, then there is no need to re-run this process, you can simply reuse the existing results. This can be obtained with:

nextflow run -resume

However, if you delete the working directory, then the results are not available anymore, and you can't resume execution.

This can be seen easily with this script:

process create_file {
    output:
		path("out.txt")

    script:
    """
    sleep 10
	echo "blabla" > out.txt
    """
}

workflow {
	create_file().view()
}

Execute it a first time with

nextflow run test.nf

it takes 10 seconds to run, then if you run

nextflow run test.nf -resume

it finishes immediately (and you can notice that the working directory name stays the same, and you get a message that it's cached). If you delete that file out.txt and re-run, it will take 10 seconds again.

This is not something that you want to do. Nextflow processes are intended to be run independently and isolated from each other from inside the working directory. On a shared filesystem, you can obviously read from and write to files outside of this directory, but this of course would not be possible if you later decided to use the AWS Batch or Google Cloud executors for example. So to ensure your workflow is portable and can be run in the cloud or locally, just make sure to only ever read from files that have been staged into your process working directory (as defined in your input block) and avoid writing to files outside of the process working directory.

Note that the publishDir directive is entirely optional. If you're just starting out with Nextflow, you can ignore the publishDir directive until you're ready to decide on what files it is that you would like your workflow to publish. Note also that only files declared in the output block can be published to the publishDir.

params.hg38genome = '/Users/name/Downloads/NM.fasta'
params.outdir = './results_new/'

process create_file {

    publishDir "${params.outdir}/check", mode: 'copy'
    debug true

    input:
    path fasta

    output:
    path "out.txt"

    script:
    """
    echo "staged files:"
    ls -1 "${fasta}"

    touch out.txt
    """
}

workflow {

    hg38genome = file( params.hg38genome )

    create_file( hg38genome )

    create_file.out.view()
}

Results:

$ nextflow run main.nf 
N E X T F L O W  ~  version 23.04.1
Launching `main.nf` [sad_franklin] DSL2 - revision: d6d4c2b069
executor >  local (1)
[44/354fa4] process > create_file [100%] 1 of 1 ✔
/path/to/work/44/354fa4771aba0090a74332c0a414ad/out.txt
staged files:
NM.fasta

The 'process working directory' in this example is:

/path/to/work/44/354fa4771aba0090a74332c0a414ad

Note also that Nextflow creates a number of 'dot' files inside this directory:

$ ls -ga --time-style='+' /path/to/work/44/354fa4771aba0090a74332c0a414ad/
total 32
drwxr-xr-x 2 users 4096  .
drwxr-xr-x 3 users 4096  ..
-rw-r--r-- 1 users    0  .command.begin
-rw-r--r-- 1 users    0  .command.err
-rw-r--r-- 1 users   23  .command.log
-rw-r--r-- 1 users   23  .command.out
-rw-r--r-- 1 users 3132  .command.run
-rw-r--r-- 1 users   69  .command.sh
-rw-r--r-- 1 users    1  .exitcode
lrwxrwxrwx 1 users   73  NM.fasta -> /Users/name/Downloads/NM.fasta
-rw-r--r-- 1 users    0  out.txt