Page 1 of 2 12 LastLast
Results 1 to 10 of 18

Thread: bash if statement complex search logic

  1. #1
    Join Date
    Jan 2014
    Beans
    22

    bash if statement complex search logic

    I have bitten off more than I can chew with some bash scripting involving a very poorly laid out log file that I have no option to change. I have 2 different scenarios that have me stumped.

    The log file is in the format of Time/Date related lines with other junk mixed in, followed by other lines detailing information about the processes run state that does NOT have date/time stamps.

    Pseudo code for the self descriptive 1st scenario, this one is easier but still has me stumped at my padawan bash scripting level:

    if logfile.log contains any line with the string 07/24/14 followed on the very next line by the text "Parsing Complete"
    then
    echo "parse complete"
    else
    echo "parse failed or did not run"

    The second scenario is similar but harder. It has the date/time stamp line followed by task states on lines following, but POTENTIAL fixes to problems after a search. I am completely lost on how to even start with this.

    Here is a sample log section:

    Process run started on 7/24/14 9:23AM
    status t8 succeed
    status t9 succeed
    status t10 succeed
    status t11 failure
    status t12 succeed

    Pseudo code:

    if logfile.log contains any line with the string 07/24/14 followed by a GROUP of lines all containing the text "status t" and the text "failure"
    then
    set a variable to the 1 or 2 digit number of the failure lines

    My problem is I just don't know how to chop the data up into pieces I can deal with. 2 lines as a group in the first scenario and a grouped chunk of lines in the 2nd scenario.

  2. #2
    Join Date
    Apr 2012
    Beans
    7,256

    Re: bash if statement complex search logic

    Often with pattern matching problems, it's as important to consider what you don't want to match - remember you are trying to find the minimal set of attributes that uniquely defines the thing you want to extract. And what's the overall structure of the log? Are blocks separated by blank lines?

    Since you have two quite distinct things, you might want to consider separating them into their own files for further processing. For example, you could do a multiline match like

    Code:
    pcregrep -nM '^.*\d{1,2}/\d{1,2}/\d{1,2}.*(\nstatus.*)+' yourlog
    (I've used pcregrep - available from the repository; you may be able to use grep in PCRE mode with the -z option, but I couldn't make the -n switch work in that case) which will give you an output like

    Code:
    17:Process run started on 7/24/14 9:23AM
    status t8 succeed
    status t9 succeed
    status t10 succeed
    status t11 failure
    status t12 succeed
    32:Process run started on 7/24/14 9:23AM
    status t8 succeed
    status t9 succeed
    status t10 succeed
    status t11 failure
    status t12 succeed
    i.e. extracts each of the "type II" blocks, prepended with their starting line numbers in the original file - you could then further process these to find the 'failure' lines

    Code:
    pcregrep -nM '^.*\d{1,2}/\d{1,2}/\d{1,2}.*(\nstatus.*)+' mylog | sed -r 's/(^[0-9]+):.*/\n\1/' > mylogII
    
    awk -vRS= -F\n '{n=$1; for(i=2;i<=NF;i++) {if ($i ~ /failure/) print n+i-1;} }' mylogII
    Just some ideas to get you started

  3. #3
    Join Date
    Aug 2011
    Location
    47°9′S 126°43W
    Beans
    2,172
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: bash if statement complex search logic

    You can transform your input with sed or awk.

    For the first case just remove the linefeed characted at the end of any line containing '7/24/14' to make it one single line with the next one. Then keep the lines with '7/24/14' and see those that contain "parsing complete"

    For the second drop the linefeed just before "status" to splice into one line the data and the varoius status reports, then keep the line with '7/24/14' and may be split them again.

    Or you can just read your file and implement the logic with a bit of Perl or Python. That could be more maintainable in the end, and not necessarily much slower. Bash one-liners are fun, but one should be pragmatic at times.
    Warning: unless noted otherwise, code in my posts should be understood as "coding suggestions", and its use may require more neurones than the two necessary for Ctrl-C/Ctrl-V.

  4. #4
    Join Date
    Jan 2014
    Beans
    22

    Re: bash if statement complex search logic

    Ok... I have still been mulling this one over and I think I have thought of an easier way to crack this nut but it is still beyond my skill level and I need help pulling it off. I can do everything I need to do with this file if I do the following FIRST.

    Scan the file line by line for date and time stamps embedded in the line. Set a variable with the date and time stamp that is found, put the date and time stamp at the beginning of each line including and after that line until the next line with a date and time stamp is found.

  5. #5
    Join Date
    Apr 2012
    Beans
    7,256

    Re: bash if statement complex search logic

    You would probably get better answers if you explained your end goal and and gave a minimal representative sample of the actual log file and what you want to extract from it - otherwise it's easy to fall into the X-Y problem trap

  6. #6
    Join Date
    Jul 2007
    Location
    Poland
    Beans
    4,499
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: bash if statement complex search logic

    Code:
    $ awk '/Process.*:[0-9][0-9][AP]M/{ d=$(NF-1)" "$NF; next; } { print d": " $0; }' <<< '17:Process run started on 7/24/14 9:23AM
    status t8 succeed
    status t9 succeed
    32:Process run started on 7/26/14 2:55PM
    status t11 failure
    status t12 succeed'
    
    7/24/14 9:23AM: status t8 succeed
    7/24/14 9:23AM: status t9 succeed
    7/26/14 2:55PM: status t11 failure
    7/26/14 2:55PM: status t12 succeed
    if your question is answered, mark the thread as [SOLVED]. Thx.
    To post code or command output, use [code] tags.
    Check your bash script here // BashFAQ // BashPitfalls

  7. #7
    Join Date
    Jan 2014
    Beans
    22

    Re: bash if statement complex search logic

    I have the examples in the first post of the log file. I can't really give examples of the end goal because even I don't know what those are. Over the next few weeks I'm going to need to do a ton of work from this file and I just haven't been able to parse it in any useful way so far.

    This particular log file is VERY poorly laid out as I state in the first email. It is generated from some very old windows software that is no longer supported. But basically it is in the format of:

    Something started on date time
    info about the task with no date or time
    info about the task with no date or time
    ""

    The number of information lines varies per task and can be any mix of successes or failures.

    I see vaphell's response and I had not thought of doing it that way... I was in the process of testing with a while read line loop. Could someone break down what is going on it vaphell's one liner? I really am in this for the learning.. not just getting the task done although I am killing 2 birds with one stone.

  8. #8
    Join Date
    Jul 2007
    Location
    Poland
    Beans
    4,499
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: bash if statement complex search logic

    in that awk code there are 2 blocks
    the 1st one tests for the 'Process ... <date>', sets the variable assuming that the timestamp is stored in the last 2 fields: $(NF-1) and $NF (NF obviously being the internal variable storing the number of fields in the current line), gets next line right off the bat ignoring block #2
    the 2nd block is reached for every other type of line and prints date: original content
    Last edited by Vaphell; August 26th, 2014 at 05:01 PM.
    if your question is answered, mark the thread as [SOLVED]. Thx.
    To post code or command output, use [code] tags.
    Check your bash script here // BashFAQ // BashPitfalls

  9. #9
    Join Date
    Jan 2014
    Beans
    22

    Re: bash if statement complex search logic

    OMG.. I have spent all day beating on this trying to figure out what witchcraft you were using that made this work on my test file but not on my live file if I changed ANYTHING about the regex portion or what portion of the file was doing something different than what I thought it was that I was breaking if I altered it.

    It wasn't witchcraft. If is a difference in a file created on windows versus my test file created on my linux box. :<

    I finally copied the ENTIRE contents of my test file into the shell of the live file AND IT STILL DIDN'T WORK. This one, I'll have to figure out tomorrow. My head hurts. One single line has kicked my butt today.

  10. #10
    Join Date
    Jul 2007
    Location
    Poland
    Beans
    4,499
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: bash if statement complex search logic

    care to post a snippet from original file, carriage returns and whatnot included?
    if your question is answered, mark the thread as [SOLVED]. Thx.
    To post code or command output, use [code] tags.
    Check your bash script here // BashFAQ // BashPitfalls

Page 1 of 2 12 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •