Bash: How can I pass a wildcard as a parameter within a script?

[Paddy Landau]

In a Bash script:

I need to pass an exclude option with a wildcard to rsync. The twist is that sometimes the wildcard needs to be omitted.

Without a wildcard, this is easy:

if some_condition; then EXCLUDE="--exclude='/data/'"; else EXCLUDE=''; fi
rsync ... ${EXCLUDE} ...

If some_condition is true, rsync gets --exclude='/data/', otherwise there's no exclude option.

When it comes to wildcards, it's more complicated. You need to turn off globbing as follows otherwise Bash expands it.

if some_condition; then EXCLUDE="--exclude='/*'"; else EXCLUDE=''; fi
set -o noglob
rsync ... ${EXCLUDE} ...
set +o noglob

I would have thought that this would work. However, when I run this in real life, rsync always acts as if EXCLUDE='' even when some_condition is true.

I have checked that EXCLUDE has been correctly set.

If I hard-code --exclude='/*' instead of using ${EXCLUDE}, then rsync works correctly.

What am I doing wrong?

[dinkidonk]

Have you tried to use "$EXCLUDE" instead of "${EXCLUDE}"?

In your example, the "*" makes little sense. Would have made sense if you would want to exclude all files with a specific extension, like "*.txt" or something.

[sudodus]

1. I did not check ,but did you try "${EXCLUDE}" (to double-quote it)?

2. Otherwise, why not 'hard-code' it in the script? You know already that it works that way (I mean to have the rsync command inside the if statement).

[scorp123]

Why not work with regular expressions instead so the exclude stays the same and files thus get selected based on whether or not they match the regular expression?
Or what exactly is that mysterious "some_condition" that you are setting?

Alternative idea: "for" loop with a "find" that uses negative file selection?

for interestingfile in `find /path/to/my/files -type f \( -iname \*.stuff -o iname \*.morestuff -o -iname \*.otherstuff \) -not -path "/path/to/stuff/that/is/not/interesting/*" `
do
  if file -i $interestingfile | grep -q text
  then
    echo $interestingfile": is a text file"
    rsync -avz --progress $interestingfile remoteuser@remotesystem:/path/where/textfiles/go
  else
    echo $interestingfile": is a binary"
    rsync -avz --progress $interestingfile remoteuser@remotesystem:/path/where/binary/stuff/goes
  fi
done

[Paddy Landau]

Have you tried to use "$EXCLUDE" instead of "${EXCLUDE}"?

In Bash, using or not using curly brackets is identical, unless there is an ambiguous situation, in which curly brackets are required. (This is an unambiguous situation, so either way would work.)

In your example, the "*" makes little sense. Would have made sense if you would want to exclude all files with a specific extension, like "*.txt" or something.

It makes perfect sense; the rsync command has been thoroughly tested. The exclusions "/" and "/*" are not the same. There are several other includes and excludes given to rsync, to give a little context.

[Paddy Landau]

1. I did not check ,but did you try "${EXCLUDE}" (to double-quote it)?

Yes. It makes no difference.

2. Otherwise, why not 'hard-code' it in the script? You know already that it works that way (I mean to have the rsync command inside the if statement).

Because it's a complex rsync with several other options, and I wish to avoid duplication (that's one way that bugs creep in over time).

[Paddy Landau]

Why not work with regular expressions instead…

Sorry, I'm not entirely sure what you are suggesting. The rsync transfers a large amount of data in quite a deep level of folders. The condition determines which folders to transfer, and while most of the options are the same in all cases, there are just a few that differ.

Does that answer your question?

EDIT: Multiple calls will cause a lot of overhead on the server, which is quite slow. So, a single rsync call is preferred.

[Paddy Landau]

I've figured out a workaround: Use --exclude-from.

I create a temporary file, and either leave it empty or write "/*" to it depending on what's needed.

So…

EXCLUDE_FILE=$( mktemp --tmpdir=${XDG_RUNTIME_DIR} z-XXXXXXXXXX )
trap "rm --force '${EXCLUDE_FILE}'" RETURN

some_condition && printf '/*\n' >"${EXCLUDE_FILE}"

rsync ... -exclude-from="${EXCLUDE_FILE}" ...

Thanks for your help and ideas.

[sudodus]

Sometimes, when it is hard to control where the wild-card is expanded, you can work around the problem by writing to a temporary file, in this case for example a 'one-liner' command with rsync, and let the main shellscript call this one-liner.

Edit: after posting I found that you have already found a[nother] workaround

[Paddy Landau]

Sometimes, when it is hard to control where the wild-card is expanded, you can work around the problem by writing to a temporary file, in this case for example a 'one-liner' command with rsync, and let the main shellscript call this one-liner.

Edit: after posting I found that you have already found a[nother] workaround

Yes, thank you, that would have been a good idea had there not been another workaround. There's always a chance of a catastrophic problem with such an approach, of course, but sometimes you don't have a choice!