Help needed with file sorting

[Merrattic]

I have some code snippets that can do what I want, but would like to be able to combine them together. I have a large number of files backed up (@ 250gb) that need sorting by file type. I would like to be able to search the whole fileset, which is many many sub directories deep, for a particular filetype, and then move them to a new location.

1st snippet

EXTS="jpg"; for ext in $EXTS; do mkdir $ext; mv  *.${ext} $ext; done

This will find all files with extension jpg and create a directory before moving them all over. (It is possible to add more extensions to the variable to further automate, but I will probably be happy to run one at a time). I need to avoid overwriting duplicates and I understand I can add a switch to mv for this:

2nd snippet

mv --backup=numbered

My problem is that this 1st snippet only covers the first directory down. I know about using find to move all files to a different location:

3rd snippet

find ~/[target folder] -type f -exec mv {} ~/[destination folder] \;

How do I go about combining, or using the intent from the 3 snippets to achieve what I am after. Also, for testing purposes, how would the same thing be done using cp?

Thanks in anticipation

[drmrgd]

A couple clarifications: do you want to move (let's say) all .jpgs to 1 directory called (let's say) JPGs, or do you need more segmentation than that (e.g JPGS/FromJuly, JPGS/MyWedding/, etc.)? Also, do you want to combine all of the .jpgs first, and then selectively pull a few out from time to time, or am I misunderstanding your second statement goal:

I would like to be able to search the whole fileset, which is many many sub directories deep, for a particular filetype, and then move them to a new location.

I think the find command will be what you want here as it seems your directory structure is too complex for a sane bash loop like in the first example. If it's the most simple of the cases, I think something along the lines of :

mkdir <dest_folder> && find <target_root_folder> -type f -iname *.jpg -exec echo mv {} /path/to/<dest_folder> \;

If you add 'echo' before the move command (or copy if you prefer), you can see what would happen with that find and move command. If you're happy with the results, simply remove 'echo' and it'll process the files.

Give us just a little more detail on your goals, and I'm sure this can be very easily simplified for you.

[papibe]

Hi Merrattic.

You almost got it. Try this.

find /path/to/source/ -type f -iname '*.jpg' \
    -exec echo mv -f --backup=numbered '{}' /path/to/destination/ \;

Same as drmrgd's suggestion, it is not actually moving files but echoing the mv command. Test it debug it, and when you feel confident remove the 'echo'.

'cp' supports the same options, so in case you prefer copy the files, just replace 'mv' with 'cp'.

Also make sure that '/path/to/destination/' in not under '/path/to/source' or you'll get in trouble.

It could aslso be valuable to include *.jpeg files too:

find ... \( -iname '*.jpg' -or -iname '*.jpeg' \) ...

Tell us how it goes.
Regards.

[Merrattic]

Thanks to drmrgd and papibe for responses. I'll give it all a go and see how I get on.

The backup requirement is key, not so much for the .jpg extension but when I get to .html I want to avoid overwriting all those index.html files!

I do like my 1st snippet as the variable EXTS can take as many extensions as I can throw at it to improve automation, hence why I was hoping to augment the mv part of the code to cover all sub directories. I know it might be a matter of putting more *'s and /'s in front of the .{$ext} part ?

[Vaphell]

1. it's easy to automate find

ext=( "*mp3" "*avi" )
find_opt=()
for e in "${ext[@]}"
do
  (( ${#find_opt[@]} )) && find_opt+=( "-o" )
  find_opt+=( "-iname" "$e" )
done
echo "${find_opt[@]}"
# -iname *mp3 -o -iname *avi
find $path \( "${find_opt[@]}" \) ....

also arrays are better than flat strings (they have fewer problems with whitespace), so even if you stay with your for loop
use

ext=("ext1" "ext2")
for e in "${ext[@]}"; do mv ...; done

2. you can enable globstar in bash
it allows for ** which matches dir chains of any length

shopt -s globstar
for f in **/*.txt; do echo $f; done

[Merrattic]

Aha, thanks for your input vaphell. So if I

ext=("ext1" "ext2")
for e in "${ext[@]}"; do mv ...; done

shopt -s globstar
EXTS=("jpg"); for ext in "${EXTS[@]}"; do mkdir $ext; mv --backup=numbered **/*.${ext} $ext; done

I'll get a directory called jpg full of every jpg in the sub directories ? (I understand I could replace "mv --backup=numbered" with "echo" just to test?)

I was missing the glob part when I tried it before....