PDA

View Full Version : Processig a csv file line by line in bash



dunryc
September 18th, 2009, 10:14 AM
Hi hoping someone can help I'm trying to process a CSV file line by line in bash the format of which is below :-


"Mr" "john" "smith" "457" "ripple road" ,"sometown","somecounty","sk11qw","2085853618",,"7000000722",,,


What I want to be able to accomplish is grabbing the same "columns" from the csv file into variables. I believe i can read the csv file line by line using the code blow



cat myfile.csv | while read line; do

#here is where i need to get the fist name and mobile number into to variables

done

So if any one has any answers that would help me i would be extremely grateful

thanks in advance Dunryc

dwhitney67
September 18th, 2009, 12:40 PM
Would this work?


#!/bin/bash

cat myfile.csv | while read line
do
echo $line | cut -d'"' -f4 # get the first name
echo $line | cut -d'"' -f18 # get the telephone number
done


P.S. This also may assist you:


#!/bin/bash

declare -i index
index=1

cat myfile.csv | while read line
do
while [ $index != 24 ]
do
echo -n "$index: "
echo $line | cut -d'"' -f$index
index=$index+1
done
done

ghostdog74
September 18th, 2009, 02:56 PM
Hi hoping someone can help I'm trying to process a CSV file line by line in bash the format of which is below :-




What I want to be able to accomplish is grabbing the same "columns" from the csv file into variables. I believe i can read the csv file line by line using the code blow



So if any one has any answers that would help me i would be extremely grateful

thanks in advance Dunryc

assume column 5 is telephone number


awk -F"," '
{
column1 = $1
tel = $5
split(column1,a," ")
print "first name: " a[1],a[2]
print "telephone: " tel
}' file

output


# more file
"Mr" "john" "smith" "457" "ripple road" ,"sometown","somecounty","sk11qw","2085853618",,"7 000000722"

# ./shell.sh
first name: "Mr" "john"
telephone: "2085853618"

ghostdog74
September 18th, 2009, 03:11 PM
Would this work?


#!/bin/bash

cat myfile.csv | while read line
do
echo $line | cut -d'"' -f4 # get the first name
echo $line | cut -d'"' -f18 # get the telephone number
done



few points
1) no need to use cat.
2) you are using cut 2 times for each line --> inefficient
3) "cut" is programmed to take in a file as argument, at the same time, internally its doing line by line processing of the file. therefore, the above can be shortened


cut -d'"' -f4,18 --output-delimiter=" " file

hal10000
September 18th, 2009, 03:12 PM
I suppose the field separator between the particular fields is a comma, then this also may work for you


OIFS=$IFS
IFS=, #notice: this is your field separator
while read var1 var2 var3 var........
# in the upper line as much variables as you need
# if th read command contains less variables than your csv-file contains fields,
# then the last declared variable contains the whole rest of the line

do
echo $var1, $var4, $var6

# note: the upper line is just an example to print out the fist, fourth and sixth field

done <myfile.csv
IFS=$OIFS

ghostdog74
September 18th, 2009, 03:17 PM
I suppose the field separator between the particular fields is a comma, then this also may work for you


OIFS=$IFS
IFS=, #notice: this is your field separator
while read var1 var2 var3 var........
# in the upper line as much variables as you need
# if th read command contains less variables than your csv-file contains fields,
# then the last declared variable contains the whole rest of the line

do
echo $var1, $var4, $var6

# note: the upper line is just an example to print out the fist, fourth and sixth field

done <myfile.csv
IFS=$OIFS


you can put your IFS inside the while loop


while IFS="," read var1 var2 var3 var4 var5 var6
do
echo $var5
done < file

dunryc
September 18th, 2009, 04:15 PM
@ hal10000

that works great and gives me the output i need but is there any way i can work through the file line by line rather than all at one time as i want to run a command with the output ie :-


echo "Hello $var2 how are you today hope things are " | gnokii --sendsms 0$var12

Thanks every one for the heads up its been a real help

Regards Dunryc

dunryc
September 20th, 2009, 12:32 PM
Thanks for all the suggestions guys got my script working great in the end