Page 1 of 3 123 LastLast
Results 1 to 10 of 30

Thread: Bash: How can I pass a wildcard as a parameter within a script?

  1. #1
    Join Date
    May 2008
    Location
    United Kingdom
    Beans
    4,784
    Distro
    Lubuntu 18.04 Bionic Beaver

    Question Bash: How can I pass a wildcard as a parameter within a script?

    In a Bash script:

    I need to pass an exclude option with a wildcard to rsync. The twist is that sometimes the wildcard needs to be omitted.

    Without a wildcard, this is easy:
    Code:
    if some_condition; then EXCLUDE="--exclude='/data/'"; else EXCLUDE=''; fi
    rsync ... ${EXCLUDE} ...
    If some_condition is true, rsync gets --exclude='/data/', otherwise there's no exclude option.

    When it comes to wildcards, it's more complicated. You need to turn off globbing as follows otherwise Bash expands it.
    Code:
    if some_condition; then EXCLUDE="--exclude='/*'"; else EXCLUDE=''; fi
    set -o noglob
    rsync ... ${EXCLUDE} ...
    set +o noglob
    I would have thought that this would work. However, when I run this in real life, rsync always acts as if EXCLUDE='' even when some_condition is true.

    I have checked that EXCLUDE has been correctly set.

    If I hard-code --exclude='/*' instead of using ${EXCLUDE}, then rsync works correctly.

    What am I doing wrong?
    Last edited by Paddy Landau; July 2nd, 2021 at 01:33 PM.
    Always make regular backups of your data (and test them).
    Visit Full Circle Magazine for beginners and seasoned Linux enthusiasts.

  2. #2
    Join Date
    Jun 2018
    Beans
    163

    Re: Bash: How can I pass a wildcard as a parameter within a script?

    Have you tried to use "$EXCLUDE" instead of "${EXCLUDE}"?

    In your example, the "*" makes little sense. Would have made sense if you would want to exclude all files with a specific extension, like "*.txt" or something.
    Have a ubuntastic day!

  3. #3
    Join Date
    Nov 2011
    Location
    /dev/root
    Beans
    Hidden!

    Re: Bash: How can I pass a wildcard as a parameter within a script?

    1. I did not check ,but did you try "${EXCLUDE}" (to double-quote it)?

    2. Otherwise, why not 'hard-code' it in the script? You know already that it works that way (I mean to have the rsync command inside the if statement).

  4. #4
    Join Date
    May 2006
    Location
    Switzerland
    Beans
    2,904
    Distro
    Ubuntu 20.04 Focal Fossa

    Re: Bash: How can I pass a wildcard as a parameter within a script?

    Why not work with regular expressions instead so the exclude stays the same and files thus get selected based on whether or not they match the regular expression?
    Or what exactly is that mysterious "some_condition" that you are setting?

    Alternative idea: "for" loop with a "find" that uses negative file selection?

    Code:
    for interestingfile in `find /path/to/my/files -type f \( -iname \*.stuff -o iname \*.morestuff -o -iname \*.otherstuff \) -not -path "/path/to/stuff/that/is/not/interesting/*" `
    do
      if file -i $interestingfile | grep -q text
      then
        echo $interestingfile": is a text file"
        rsync -avz --progress $interestingfile remoteuser@remotesystem:/path/where/textfiles/go
      else
        echo $interestingfile": is a binary"
        rsync -avz --progress $interestingfile remoteuser@remotesystem:/path/where/binary/stuff/goes
      fi
    done

  5. #5
    Join Date
    May 2008
    Location
    United Kingdom
    Beans
    4,784
    Distro
    Lubuntu 18.04 Bionic Beaver

    Re: Bash: How can I pass a wildcard as a parameter within a script?

    Quote Originally Posted by dinkidonk View Post
    Have you tried to use "$EXCLUDE" instead of "${EXCLUDE}"?
    In Bash, using or not using curly brackets is identical, unless there is an ambiguous situation, in which curly brackets are required. (This is an unambiguous situation, so either way would work.)
    Quote Originally Posted by dinkidonk View Post
    In your example, the "*" makes little sense. Would have made sense if you would want to exclude all files with a specific extension, like "*.txt" or something.
    It makes perfect sense; the rsync command has been thoroughly tested. The exclusions "/" and "/*" are not the same. There are several other includes and excludes given to rsync, to give a little context.
    Always make regular backups of your data (and test them).
    Visit Full Circle Magazine for beginners and seasoned Linux enthusiasts.

  6. #6
    Join Date
    May 2008
    Location
    United Kingdom
    Beans
    4,784
    Distro
    Lubuntu 18.04 Bionic Beaver

    Re: Bash: How can I pass a wildcard as a parameter within a script?

    Quote Originally Posted by sudodus View Post
    1. I did not check ,but did you try "${EXCLUDE}" (to double-quote it)?
    Yes. It makes no difference.
    Quote Originally Posted by sudodus View Post
    2. Otherwise, why not 'hard-code' it in the script? You know already that it works that way (I mean to have the rsync command inside the if statement).
    Because it's a complex rsync with several other options, and I wish to avoid duplication (that's one way that bugs creep in over time).
    Always make regular backups of your data (and test them).
    Visit Full Circle Magazine for beginners and seasoned Linux enthusiasts.

  7. #7
    Join Date
    May 2008
    Location
    United Kingdom
    Beans
    4,784
    Distro
    Lubuntu 18.04 Bionic Beaver

    Re: Bash: How can I pass a wildcard as a parameter within a script?

    Quote Originally Posted by scorp123 View Post
    Why not work with regular expressions instead…
    Sorry, I'm not entirely sure what you are suggesting. The rsync transfers a large amount of data in quite a deep level of folders. The condition determines which folders to transfer, and while most of the options are the same in all cases, there are just a few that differ.

    Does that answer your question?

    EDIT: Multiple calls will cause a lot of overhead on the server, which is quite slow. So, a single rsync call is preferred.
    Always make regular backups of your data (and test them).
    Visit Full Circle Magazine for beginners and seasoned Linux enthusiasts.

  8. #8
    Join Date
    May 2008
    Location
    United Kingdom
    Beans
    4,784
    Distro
    Lubuntu 18.04 Bionic Beaver

    Re: Bash: How can I pass a wildcard as a parameter within a script?

    I've figured out a workaround: Use --exclude-from.

    I create a temporary file, and either leave it empty or write "/*" to it depending on what's needed.

    So…
    Code:
    EXCLUDE_FILE=$( mktemp --tmpdir=${XDG_RUNTIME_DIR} z-XXXXXXXXXX )
    trap "rm --force '${EXCLUDE_FILE}'" RETURN
    
    some_condition && printf '/*\n' >"${EXCLUDE_FILE}"
    
    rsync ... -exclude-from="${EXCLUDE_FILE}" ...
    Thanks for your help and ideas.
    Always make regular backups of your data (and test them).
    Visit Full Circle Magazine for beginners and seasoned Linux enthusiasts.

  9. #9
    Join Date
    Nov 2011
    Location
    /dev/root
    Beans
    Hidden!

    Re: Bash: How can I pass a wildcard as a parameter within a script?

    Sometimes, when it is hard to control where the wild-card is expanded, you can work around the problem by writing to a temporary file, in this case for example a 'one-liner' command with rsync, and let the main shellscript call this one-liner.

    Edit: after posting I found that you have already found a[nother] workaround

  10. #10
    Join Date
    May 2008
    Location
    United Kingdom
    Beans
    4,784
    Distro
    Lubuntu 18.04 Bionic Beaver

    Re: Bash: How can I pass a wildcard as a parameter within a script?

    Quote Originally Posted by sudodus View Post
    Sometimes, when it is hard to control where the wild-card is expanded, you can work around the problem by writing to a temporary file, in this case for example a 'one-liner' command with rsync, and let the main shellscript call this one-liner.

    Edit: after posting I found that you have already found a[nother] workaround
    Yes, thank you, that would have been a good idea had there not been another workaround. There's always a chance of a catastrophic problem with such an approach, of course, but sometimes you don't have a choice!
    Always make regular backups of your data (and test them).
    Visit Full Circle Magazine for beginners and seasoned Linux enthusiasts.

Page 1 of 3 123 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •