Page 1 of 3 123 LastLast
Results 1 to 10 of 22

Thread: Help needed with file sorting

  1. #1
    Join Date
    Jul 2012
    Beans
    315
    Distro
    Xubuntu 12.04 Precise Pangolin

    Help needed with file sorting

    I have some code snippets that can do what I want, but would like to be able to combine them together. I have a large number of files backed up (@ 250gb) that need sorting by file type. I would like to be able to search the whole fileset, which is many many sub directories deep, for a particular filetype, and then move them to a new location.

    1st snippet
    Code:
    EXTS="jpg"; for ext in $EXTS; do mkdir $ext; mv  *.${ext} $ext; done
    This will find all files with extension jpg and create a directory before moving them all over. (It is possible to add more extensions to the variable to further automate, but I will probably be happy to run one at a time). I need to avoid overwriting duplicates and I understand I can add a switch to mv for this:

    2nd snippet
    Code:
    mv --backup=numbered
    My problem is that this 1st snippet only covers the first directory down. I know about using find to move all files to a different location:

    3rd snippet
    Code:
    find ~/[target folder] -type f -exec mv {} ~/[destination folder] \;
    How do I go about combining, or using the intent from the 3 snippets to achieve what I am after. Also, for testing purposes, how would the same thing be done using cp?

    Thanks in anticipation
    The best things in life are free, so what are we paying for?

  2. #2
    Join Date
    Apr 2011
    Location
    Maryland
    Beans
    1,461
    Distro
    Kubuntu 12.04 Precise Pangolin

    Re: Help needed with file sorting

    A couple clarifications: do you want to move (let's say) all .jpgs to 1 directory called (let's say) JPGs, or do you need more segmentation than that (e.g JPGS/FromJuly, JPGS/MyWedding/, etc.)? Also, do you want to combine all of the .jpgs first, and then selectively pull a few out from time to time, or am I misunderstanding your second statement goal:

    I would like to be able to search the whole fileset, which is many many sub directories deep, for a particular filetype, and then move them to a new location.
    I think the find command will be what you want here as it seems your directory structure is too complex for a sane bash loop like in the first example. If it's the most simple of the cases, I think something along the lines of :

    Code:
    mkdir <dest_folder> && find <target_root_folder> -type f -iname *.jpg -exec echo mv {} /path/to/<dest_folder> \;
    If you add 'echo' before the move command (or copy if you prefer), you can see what would happen with that find and move command. If you're happy with the results, simply remove 'echo' and it'll process the files.

    Give us just a little more detail on your goals, and I'm sure this can be very easily simplified for you.

  3. #3
    Join Date
    Feb 2009
    Location
    Dallas, TX
    Beans
    6,840
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: Help needed with file sorting

    Hi Merrattic.

    You almost got it. Try this.
    Code:
    find /path/to/source/ -type f -iname '*.jpg' \
        -exec echo mv -f --backup=numbered '{}' /path/to/destination/ \;
    Same as drmrgd's suggestion, it is not actually moving files but echoing the mv command. Test it debug it, and when you feel confident remove the 'echo'.

    'cp' supports the same options, so in case you prefer copy the files, just replace 'mv' with 'cp'.

    Also make sure that '/path/to/destination/' in not under '/path/to/source' or you'll get in trouble.

    It could aslso be valuable to include *.jpeg files too:
    Code:
    find ... \( -iname '*.jpg' -or -iname '*.jpeg' \) ...
    Tell us how it goes.
    Regards.

  4. #4
    Join Date
    Jul 2012
    Beans
    315
    Distro
    Xubuntu 12.04 Precise Pangolin

    Re: Help needed with file sorting

    Thanks to drmrgd and papibe for responses. I'll give it all a go and see how I get on.

    The backup requirement is key, not so much for the .jpg extension but when I get to .html I want to avoid overwriting all those index.html files!

    I do like my 1st snippet as the variable EXTS can take as many extensions as I can throw at it to improve automation, hence why I was hoping to augment the mv part of the code to cover all sub directories. I know it might be a matter of putting more *'s and /'s in front of the .{$ext} part ?
    The best things in life are free, so what are we paying for?

  5. #5
    Join Date
    Jul 2007
    Location
    Poland
    Beans
    4,393
    Distro
    Ubuntu 10.04 Lucid Lynx

    Re: Help needed with file sorting

    1. it's easy to automate find

    Code:
    ext=( "*mp3" "*avi" )
    find_opt=()
    for e in "${ext[@]}"
    do
      (( ${#find_opt[@]} )) && find_opt+=( "-o" )
      find_opt+=( "-iname" "$e" )
    done
    echo "${find_opt[@]}"
    # -iname *mp3 -o -iname *avi
    find $path \( "${find_opt[@]}" \) ....
    also arrays are better than flat strings (they have fewer problems with whitespace), so even if you stay with your for loop
    use
    Code:
    ext=("ext1" "ext2")
    for e in "${ext[@]}"; do mv ...; done
    2. you can enable globstar in bash
    it allows for ** which matches dir chains of any length
    Code:
    shopt -s globstar
    for f in **/*.txt; do echo $f; done
    Last edited by Vaphell; November 22nd, 2012 at 03:44 PM.
    if your question is answered, mark the thread as [SOLVED]. Thx.
    To post code or command output, use [code] tags.
    Check your bash script here // BashFAQ // BashPitfalls

  6. #6
    Join Date
    Jul 2012
    Beans
    315
    Distro
    Xubuntu 12.04 Precise Pangolin

    Re: Help needed with file sorting

    Aha, thanks for your input vaphell. So if I

    ext=("ext1" "ext2")
    for e in "${ext[@]}"; do mv ...; done

    shopt -s globstar
    EXTS=("jpg"); for ext in "${EXTS[@]}"; do mkdir $ext; mv --backup=numbered **/*.${ext} $ext; done
    I'll get a directory called jpg full of every jpg in the sub directories ? (I understand I could replace "mv --backup=numbered" with "echo" just to test?)

    I was missing the glob part when I tried it before....
    Last edited by Merrattic; November 22nd, 2012 at 04:30 PM.
    The best things in life are free, so what are we paying for?

  7. #7
    Join Date
    Jul 2007
    Location
    Poland
    Beans
    4,393
    Distro
    Ubuntu 10.04 Lucid Lynx

    Re: Help needed with file sorting

    in theory exactly that should happen.
    yes, echoing first is a smart thing to do, never run mv/rm if you are not absolutely sure what will happen as there is no undo.
    if your question is answered, mark the thread as [SOLVED]. Thx.
    To post code or command output, use [code] tags.
    Check your bash script here // BashFAQ // BashPitfalls

  8. #8
    Join Date
    Jul 2012
    Beans
    315
    Distro
    Xubuntu 12.04 Precise Pangolin

    Re: Help needed with file sorting

    OK, had a chance to test. All methods work fine in terms of going down the chain of all the sub directories and moving files. The "--backup=numbered" is not working in any of the methods though so any files with the same name are getting overwritten. I tried changing the timestamp on one of the files with the same name but this didn't help. Using the -i switch works. ??
    The best things in life are free, so what are we paying for?

  9. #9
    Join Date
    Apr 2012
    Beans
    5,531

    Re: Help needed with file sorting

    are you testing with the 'echo' still in place by any chance? the --backup=numbered won't come into effect until you do it for real and there are some actual conflicts

    Code:
    $ ls -R
    .:
    dir1  dir2  dir3
    
    ./dir1:
    file.avi  file.jpg  file.mp3  file.mp4  subdir
    
    ./dir1/subdir:
    file.avi  file.jpg  file.mp3  file.mp4
    
    ./dir2:
    file.avi  file.jpg  file.mp3  file.mp4  subdir
    
    ./dir2/subdir:
    file.avi  file.jpg  file.mp3  file.mp4
    
    ./dir3:
    $
    $ mkdir {avi,jpg,mp3,mp4}
    $ shopt -s globstar; exts=(avi jpg mp3 mp4); for ext in "${exts[@]}"; do mv --backup=numbered -t "$ext/" **/*."$ext" ; done
    $
    $ ls -R
    .:
    avi  dir1  dir2  dir3  jpg  mp3  mp4
    
    ./avi:
    file.avi  file.avi.~1~  file.avi.~2~  file.avi.~3~
    
    ./dir1:
    subdir
    
    ./dir1/subdir:
    
    ./dir2:
    subdir
    
    ./dir2/subdir:
    
    ./dir3:
    
    ./jpg:
    file.jpg  file.jpg.~1~  file.jpg.~2~  file.jpg.~3~
    
    ./mp3:
    file.mp3  file.mp3.~1~  file.mp3.~2~  file.mp3.~3~
    
    ./mp4:
    file.mp4  file.mp4.~1~  file.mp4.~2~  file.mp4.~3~

  10. #10
    Join Date
    Jul 2012
    Beans
    315
    Distro
    Xubuntu 12.04 Precise Pangolin

    Re: Help needed with file sorting

    Yep, testing for real

    Ah, just spotted you have a "-t" after --backup=numbered. Let me try :~
    The best things in life are free, so what are we paying for?

Page 1 of 3 123 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •