Results 1 to 10 of 10

Thread: Reading a data from files and saving elsewhere

  1. #1
    Join Date
    May 2010
    Beans
    75
    Distro
    Ubuntu 12.04 Precise Pangolin

    Reading a data from files and saving elsewhere

    Guys, I have a list of files, say, 001.dat to 099.dat. First 3 lines of each file should not be considered. From next, the data is in two columns, X and Y column. What I want is following:
    From each file, find the value of X and Y corresponding to the two maximum values of Y. Then save those values in another file in columns of: File name, X1, Y1, X2, Y2, Y3 where Y3=(Y1+Y2)/2.
    may I ask your help in doing that?
    Last edited by sd@ksu; July 4th, 2013 at 05:55 AM. Reason: additional info

  2. #2
    Join Date
    Jul 2007
    Location
    Poland
    Beans
    4,499
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: Reading a data from files and saving elsewhere

    Code:
    for f in *.dat
    do
      sed -n '4,$p' "$f" | sort -k2nr | awk -v f="$f" 'NR<3{ x[NR]=$1; y[NR]=$2; }
                                                       END { printf( "%s %g %g %g %g %g\n", f, x[1], y[1], x[2], y[2], (y[1]+y[2])/2 ); }'
    done > results.txt
    if your question is answered, mark the thread as [SOLVED]. Thx.
    To post code or command output, use [code] tags.
    Check your bash script here // BashFAQ // BashPitfalls

  3. #3
    Join Date
    May 2010
    Beans
    75
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Reading a data from files and saving elsewhere

    Dear Vaphell: Thanks for your reply. The code does something (I am using it like a black box!) but does not give correct answers

  4. #4
    Join Date
    Jul 2007
    Location
    Poland
    Beans
    4,499
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: Reading a data from files and saving elsewhere

    give a sample of input data then and the expected result because it works on my test files
    Code:
    $ cat 001.txt
    1...
    2...
    3...
    10 22
    23 154
    33 35
    3  12
    3  124
    5  1267
    6  35
    7  357
    8   732
    
    $ cat 002.txt
    1 9999
    2 9999
    3 9999
    10 2
    23 14
    33 3
    3  1
    3  14
    5  167
    6  3
    7  37
    8   32
    
    $ for f in *.txt
    > do
    >   sed -n '4,$p' "$f" | sort -k2nr | awk -v f="$f" 'NR<3{ x[NR]=$1; y[NR]=$2; }
    >                                                    END { printf( "%s %g %g %g %g %g\n", f, x[1], y[1], x[2], y[2], (y[1]+y[2])/2 ); }'
    > done
    001.txt 5 1267 8 732 999.5
    002.txt 5 167 7 37 102
    if your question is answered, mark the thread as [SOLVED]. Thx.
    To post code or command output, use [code] tags.
    Check your bash script here // BashFAQ // BashPitfalls

  5. #5
    Join Date
    May 2010
    Beans
    75
    Distro
    Ubuntu 12.04 Precise Pangolin

    Lightbulb Re: Reading a data from files and saving elsewhere

    I have uploaded two files here:
    http://ubuntuone.com/5DZi50gEgX6i2xD3K813vQ
    http://ubuntuone.com/1Ipb7oP2LK0l8AmIwIo2Nq

    One thing I said wrongly in my first post, that, the first FOUR lines should be omitted. but i have tried your code by manually deleting one line from top but it still does not work on my system for some reason. May be, it has something to do with the way the data is displayed. Please see if something could be done about it. Thanks a lot Vaphell.

  6. #6
    Join Date
    Apr 2012
    Beans
    6,933

    Re: Reading a data from files and saving elsewhere

    I think the difference is you didn't specify that the values to be sorted would be in scientific notation - Until Vaphell checks back in, I might suggest changing

    Code:
    sort -k2nr
    to

    Code:
    sort -k2gr
    See if that works - also no need to delete the first line, just change 4,$ to 5,$ in the sed range

  7. #7
    Join Date
    May 2010
    Beans
    75
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Reading a data from files and saving elsewhere

    Sure, I shall try that. I think that would work . I see what you mean. Thanks steeldriver.
    .
    PS: When I said manually deleting one line from top, I did not put it correctly. I meant deleting one line from top in the data file, not the code.

  8. #8
    Join Date
    May 2010
    Beans
    75
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Reading a data from files and saving elsewhere

    Thanks to you both, the code works!!! I have changed it to the following ( for displaying in scientific notation and also preserving the number of significant digits in my calculation:


    for f in *.chi
    do
    sed -n '5,$p' "$f" | sort -k2gr | awk -v f="$f" 'NR<4{ x[NR]=$1; y[NR]=$2; }
    END { printf( "%s %e %e %e %e %e\n", f, x[1], y[1], x[2], y[2], (y[1]+y[2])/2 ); }'
    done > results.txt
    It displays the results correctly. Thanks to both of you for saving so much of my time. I should improve my scripting skills!!
    For the calculation part of "(y[1]+y[2]/)/2", do you think the number displayed is correct? I mean, considering the number of significant figures...it seems ok to me and hopefully am not missing anything elementary here.
    Last edited by sd@ksu; July 5th, 2013 at 04:35 PM. Reason: Solved

  9. #9
    Join Date
    May 2010
    Beans
    75
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Reading a data from files and saving elsewhere

    How do I mark this as solved?

  10. #10
    Join Date
    Jul 2007
    Location
    Poland
    Beans
    4,499
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: Reading a data from files and saving elsewhere

    https://wiki.ubuntu.com/UnansweredPo.../SolvedThreads

    btw you should have left the NR<3 as it was. Sed is where the header lines are removed. NR<3 in awk is for reading top 2 results from the sorted list (row 1 and 2). NR<4 means you read 3 top results but x[3] and y[3] are not used anywhere, so the code still works fine, just does one completely useless step.

    sed: remove header lines -> sort: numeric sort in descending order -> awk: get top 2 results and do stuff with them
    if your question is answered, mark the thread as [SOLVED]. Thx.
    To post code or command output, use [code] tags.
    Check your bash script here // BashFAQ // BashPitfalls

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •