Getting files from BaseMount (Using find command to loop through the symlinks)

Question

[SaltedPork]

I'm using basemount on centos and want to download all the fastq.gz files. I want to use the find command to locate each zipped file and then download them in a bash loop. The problem is that the command (below) only returns File system loop detected errors, and Too many symlinks errors. Any ideas?

find -L . -type f -name 'fastq.gz'

Answer 1

[steve]

I would suggest using rsync instead of a combination of find and cp. I have a workflow description here. The gist:

first use find to make a list of files to be copied

find "${basepace_projects}/${project_ID}" -name "*.fastq.gz" | sed -e "s|${basepace_projects}/||g" > "$project_files_list"

use that list as a filter pattern file for rsync (check the output with --dry-run first)

rsync --progress -vRt --files-from="$project_files_list" "$basepace_projects" "$output_directory"

Answer 2

[Paul]

Hi, what about command: find path_bs -maxdepth 10 -name "*fastq.gz" -exec cp {} path/to/save \; Does it works?

[SaltedPork]

Hi, yes this is good, my mistake was using a . instead of the full path. I also used the exec command you used and it worked, so many thanks.

[GenoMax]

I moved this comment to an answer. You can accept (green check mark) it.

[Paul]

I was happy to help!

Answer 3

[andrew.j.skelton73]

cp `find ./ -type f -name "*.fastq.gz"` ~/My_Fastq

[GenoMax]

Does that command traverse/read symlinks?

[andrew.j.skelton73]

When using basemount, it works no problem for me to retrieve fastq. AFAIK, basemount isn't (or shouldn't be symlinking), the files should appear normally, but are only fully retrieved from base space when a command is carried out on them. I wonder if invoking the -L flag in find is treating them all as symlinks, based on the way that basemount works?

[SaltedPork]

I have also read that BaseMount isn't using symlinks, and yet when I use -L it worked, not sure what's going on there because they look like symlinks to me (in my terminal).