Page 2 of 2 FirstFirst 12
Results 11 to 20 of 20

Thread: grep and curl won't play nice together :/

  1. #11
    Join Date
    Apr 2012
    Beans
    5,704

    Re: grep and curl won't play nice together :/

    it shouldn't matter how you provide the input to grep - you can save the curl output to a shell variable if you want
    Code:
    source="$(curl -s https://api.twitch.tv/kraken/streams)"
    then either
    Code:
    echo "$source" | grep -Eo '"https?:[^"]*streams[^"]*"'
    or (using a bash 'here string')
    Code:
    grep -Eo '"https?:[^"]*streams[^"]*"' <<< "$source"

  2. #12
    Join Date
    Aug 2014
    Beans
    9

    Re: grep and curl won't play nice together :/

    Yeah, I guess I still have much to learn steel. Working.


    Code:
    #!/bin/bash
    
    clear
    #GENERAL VARIABLES
    confdir=$(pwd | sed 's/scripts//g') #config directory
    logdir="$confdir"logs"" #logs directory
    #
    
    function featured() {
    twkeywords=$(grep "twkeywords=" $confdir/config | sed 's/twkeywords=//g')
    source=$(sudo curl -s https://api.twitch.tv/kraken/streams)
    streams=$(echo "$source" | grep -Eo '"https?:[^"]*streams[^"]*"' | cut -d / -f 6 | tr '"' ' ' | head -n -5)
    
    
    
    for i in ${twkeywords[@]}; do
        for x in ${streams[@]}; do    
            if [ "$i" == "$x" ]; then
                found="$found $i"
            fi
        done
    clear
    printf " $x = $i : $found"
    done
    
    found=""
    
    }
    featured

  3. #13
    Join Date
    Jul 2007
    Location
    Poland
    Beans
    4,425
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: grep and curl won't play nice together :/

    by accident if anything

    Code:
    for i in ${twkeywords[@]}; do
        for x in ${streams[@]}; do
    this piece calls 2 arrays...

    Code:
    twkeywords=$(grep "twkeywords=" $confdir/config | sed 's/twkeywords=//g')
    streams=$(echo "$source" | grep -Eo '"https?:[^"]*streams[^"]*"' | cut -d / -f 6 | tr '"' ' ' | head -n -5)
    ... but these variables are not arrays
    if your question is answered, mark the thread as [SOLVED]. Thx.
    To post code or command output, use [code] tags.
    Check your bash script here // BashFAQ // BashPitfalls

  4. #14
    Join Date
    Aug 2014
    Beans
    9

    Re: grep and curl won't play nice together :/

    Quote Originally Posted by Vaphell View Post
    by accident if anything

    Code:
    for i in ${twkeywords[@]}; do
        for x in ${streams[@]}; do
    this piece calls 2 arrays...

    Code:
    twkeywords=$(grep "twkeywords=" $confdir/config | sed 's/twkeywords=//g')
    streams=$(echo "$source" | grep -Eo '"https?:[^"]*streams[^"]*"' | cut -d / -f 6 | tr '"' ' ' | head -n -5)
    ... but these variables are not arrays
    At this point of my knowledge of Bash, I couldn't tell ya'. Especially when talking about arrays in Bash. Their definitely not declared . But hey, it works and I'm just poking around testing things.

    In this case:
    $twkeywords=giantwaffle lirik swiftor cohcarnage dansgaming summit1g day9tv manvsgame ezekiel_iii. I don't want to announce all the top streams, just the ones I follow. Which is listed in a config file, something like a following list. If you copy pasta, it wouldn't work.

    $streams=
    roomonfire
    joindotared
    nightblue3
    imaqtpie
    lirik
    reckful
    ceh9
    flosd
    gsstudio_dota
    forsenlol
    ihearthusite
    sevadus
    oxichampion
    manvsgame
    timthetatman
    kolento
    starladder1
    riotgames
    quickybaby
    debitorlp
    mdstephano
    lightofheaven
    fishplaystreetfighter
    sethbling
    diablo

    They have the same format or style, so I thought that array command would do.

  5. #15
    Join Date
    Jul 2007
    Location
    Poland
    Beans
    4,425
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: grep and curl won't play nice together :/

    Code:
    #!/bin/bash
    
    readarray -t streams < <( curl -s https://api.twitch.tv/kraken/streams \
                              | grep -Eo 'https?:[^"]*streams[^"]*' \
                              | awk -F/ '{ print $NF; }' \
                              | grep -Ff streams.list )
    
    printf 'stream "%s" found\n' "${streams[@]}"
    Code:
    $ cat streams.list
    giantwaffle
    lirik
    swiftor
    cohcarnage
    dansgaming
    summit1g
    day9tv
    manvsgame
    ezekiel_iii
    $ ./streams.bash
    stream "day9tv" found
    stream "swiftor" found
    stream "dansgaming" found
    if your question is answered, mark the thread as [SOLVED]. Thx.
    To post code or command output, use [code] tags.
    Check your bash script here // BashFAQ // BashPitfalls

  6. #16
    Join Date
    Jul 2007
    Location
    Poland
    Beans
    4,425
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: grep and curl won't play nice together :/

    triple post due to killer lag
    Last edited by Vaphell; August 22nd, 2014 at 04:13 AM.
    if your question is answered, mark the thread as [SOLVED]. Thx.
    To post code or command output, use [code] tags.
    Check your bash script here // BashFAQ // BashPitfalls

  7. #17
    Join Date
    Jul 2007
    Location
    Poland
    Beans
    4,425
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: grep and curl won't play nice together :/

    triple post due to killer lag
    Last edited by Vaphell; August 22nd, 2014 at 04:14 AM.
    if your question is answered, mark the thread as [SOLVED]. Thx.
    To post code or command output, use [code] tags.
    Check your bash script here // BashFAQ // BashPitfalls

  8. #18
    Join Date
    Aug 2014
    Beans
    9

    Re: grep and curl won't play nice together :/

    Quote Originally Posted by Vaphell View Post
    Code:
    #!/bin/bash
    
    readarray -t streams < <( curl -s https://api.twitch.tv/kraken/streams \
                              | grep -Eo 'https?:[^"]*streams[^"]*' \
                              | awk -F/ '{ print $NF; }' \
                              | grep -Ff streams.list )
    
    printf 'stream "%s" found\n' "${streams[@]}"
    Thanks vaphell. I had no idea there was even a readarray command. Also I know of <, and <<, and <<<, but not < <(). Is that the syntax to inject a function into a read command or something else?

  9. #19
    Join Date
    Jul 2007
    Location
    Poland
    Beans
    4,425
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: grep and curl won't play nice together :/

    readarray is a bash way to create arrays from inputs on per line basis. I assume it was introduced because it is a relatively common use case even though the same thing should be possible with a bit more convoluted read or with a while read loop.
    Code:
    $ IFS=$'\n' read -rd '' -a arr <<< $'a a a\nb b b\nc c c\n'
    $ printf '[%s]\n' "${arr[@]}"
    [a a a]
    [b b b]
    [c c c]
    $ arr=()
    $ while read -r ln; do arr+=( "$ln" ); done <<< $'a a a\nb b b\nc c c\n'
    $ printf '[%s]\n' "${arr[@]}"
    [a a a]
    [b b b]
    [c c c]
    []
    there are some differences in how the empty lines are treated though.

    < <() is 2 separate features in reality, that's why the space between <'s is important
    first part is the standard redirection from a file <, the other is called a process substitution - it takes a cmd output and makes a pseudofile out of it. Whenever a file param is expected, you can use <(...) there.

    Code:
    $ grep -f <( echo bcd ) <<< "abcdef"
    abcdef
    there is also version in the other direction: > >(...), eg used with exec to pass all script outputs through some program.
    Last edited by Vaphell; August 23rd, 2014 at 11:47 PM.
    if your question is answered, mark the thread as [SOLVED]. Thx.
    To post code or command output, use [code] tags.
    Check your bash script here // BashFAQ // BashPitfalls

  10. #20
    Join Date
    Aug 2014
    Beans
    9

    Re: grep and curl won't play nice together :/

    Quote Originally Posted by Vaphell View Post
    readarray is a bash way to create arrays from inputs on per line basis. I assume it was introduced because it is a relatively common use case even though the same thing should be possible with a bit more convoluted read or with a while read loop.
    Code:
    $ IFS=$'\n' read -rd '' -a arr <<< $'a a a\nb b b\nc c c\n'
    $ printf '[%s]\n' "${arr[@]}"
    [a a a]
    [b b b]
    [c c c]
    $ arr=()
    $ while read -r ln; do arr+=( "$ln" ); done <<< $'a a a\nb b b\nc c c\n'
    $ printf '[%s]\n' "${arr[@]}"
    [a a a]
    [b b b]
    [c c c]
    []
    there are some differences in how the empty lines are treated though.

    < <() is 2 separate features in reality, that's why the space between <'s is important
    first part is the standard redirection from a file <, the other is called a process substitution - it takes a cmd output and makes a pseudofile out of it. Whenever a file param is expected, you can use <(...) there.

    Code:
    $ grep -f <( echo bcd ) <<< "abcdef"
    abcdef
    there is also version in the other direction: > >(...), eg used with exec to pass all script outputs through some program.
    Gotcha! Thank you so much so a understandable explanation! Just when you thought you had a slight understanding of something... redirection. Mind blown .

Page 2 of 2 FirstFirst 12

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •