Page 1 of 2 12 LastLast
Results 1 to 10 of 14

Thread: Automated Downloading of Radio Show Files

  1. #1
    Join Date
    Nov 2018
    Beans
    12

    Automated Downloading of Radio Show Files

    I tried to do a search for wget, but got an error message. Forgive me if I've posted this before or on other forums (fora?).



    I run a community radio station that airs radio shows from all over the world. The audio files for these come in via RSS, Dropbox, emailed links, and by visits to private servers. The RSS-fed shows handle themselves, as do the ones distributed via Dropbox (mostly). The ones that take up the most time week in and week out are the ones where I have to manually go to a website, click on links, then download the associated files.
    We air one show that airs every day Monday through Friday; each show comes in three parts. I retrieve the (mp2) files by logging into the website with username and password where I'm greeted with a list of files.


    Let's say the site is h++ps://zzzzz.zz/bb/bbaudio.php



    A pop-up box allows me to put in my username and password, after which I see a list of available audio files, e.g.:


    3.19BB_part1.mp2
    3.19BB_part2.mp2
    3.19BB_part3.mp2


    I've tried desperately to use wget to automate the process to no avail. I've tried various versions of this:


    wget –-user=USER –-password=PASS -r -nc -e robots=off -A.mp2 h++ps://zzzzz.zz/bb/bbaudio.php -P /media/yow/shows


    I've moved the username and password later in the string, I've incorporated it in the URL, I've tried it without the options and with other options, still nada! Essentially, the server tells me to hit the bricks....
    Truth is, I'm a pretty good radio guy, but I'm terrible at coding, syntax, and magic incantations. Any thoughts on an easy-to-use, easy-to-understand system that will work? It gets really frustrating when someone says, "Did you marginalize the krelm while it was furled? Try masticating the vergle with the string \oo/*[~"applebutter"=IPs@]" You have to treat me like I am your dumb brother-in-law and not assume I know as much as I really do!

    Oh, and FWIW, Ubuntu 20.04.2 LTS.

  2. #2
    Join Date
    Mar 2011
    Location
    U.K.
    Beans
    Hidden!
    Distro
    Ubuntu 20.04 Focal Fossa

    Re: Automated Downloading of Radio Show Files

    The ones that take up the most time week in and week out are the ones where I have to manually go to a website, click on links, then download the associated files.
    Your thread title says it all .. try a dash of automation scripting to automate the slog of visiting each site to download files.
    There are several tools I use for such tasks (although not radio stations as you cite).

    Actiona (using Qt objects) is easy to use. it is in the repo.

    You can prepare an Actionascript2 list of URL's to run through to grab files.
    If you offer a few examples of sites I can start you off with an example Actiona script.

    Another automation utility I suggest is Albert. This is not in Ubuntu repo and you have to download and install.
    Then a python extension can run through a list of bookmarks or websites.

    Just two examples. I use them extensively for automation scripts.

    ===================================

    [Postscript]
    Actually browsing around I found this Pandora's box which seems relevant ...

    https://github.com/ebu/awesome-broadcasting


    and just one gem ... note reference to automation
    https://github.com/ElvishArtisan/rivendell

    Last edited by dragonfly41; March 17th, 2021 at 11:17 AM. Reason: further searching ..

  3. #3
    Join Date
    Jan 2010
    Location
    Wheeling WV USA
    Beans
    1,819
    Distro
    Xubuntu 18.04 Bionic Beaver

    Re: Automated Downloading of Radio Show Files

    one thing to be aware of is the method of keeping you logged in at a website varies. many use time signatures in certain cookies. they are not as simple as how wget does it. almost all sites can be automated but they are not all done the same. for example, youtube-dl does very different thing on many different sites. automating a site designed for human interaction is the most difficult.
    Mask wearer, Social distancer, System Administrator, Programmer, Linux advocate, Command Line user, Ham radio operator (KA9WGN/8, tech), Photographer (hobby), occasional tweeter

  4. #4
    Join Date
    Mar 2011
    Location
    U.K.
    Beans
    Hidden!
    Distro
    Ubuntu 20.04 Focal Fossa

    Re: Automated Downloading of Radio Show Files

    automating a site designed for human interaction is the most difficult.


    Agreed. That is why I explore UI automation tools like Sikulix (Java) and Selenium
    Actiona is quite easy to use and powerful little beast. It links to apps such as Tesserat.

    [P.S] Also adding Puppeteer to the list of web scraping tools.

    https://pptr.dev/
    Last edited by dragonfly41; March 17th, 2021 at 11:51 AM. Reason: added puppeteer

  5. #5
    Join Date
    Nov 2018
    Beans
    12

    Re: Automated Downloading of Radio Show Files

    This sounds like just the ticket. The question is, can an audio dude master this? I've gotten wget to successfully download files, but only from non-password-protected sites. Since I can't post username/password info, here's a sorta-kinda example (chosen at random and not a source we use) of a download site: https://www.rusc.com/old_time_radio_downloads.aspx

    FWIW, here's a screen-grab of an actual list; this is what I see after I log in with my username and password. The list changes week-to-week as new shows are added and old ones deleted:



    Thanks for any help, encouragement, recipes, suggestions you can offer!

  6. #6
    Join Date
    Mar 2011
    Location
    U.K.
    Beans
    Hidden!
    Distro
    Ubuntu 20.04 Focal Fossa

    Re: Automated Downloading of Radio Show Files

    Perhaps an easy experiment to start with is installing Klipper in Chromium browser

    https://chrome.google.com/webstore/d.../related?hl=en

    Visit your site and scrape list of downloads into CSV.

    As I see it (with no experience in your field), it is easy to grab all downloads but you will want to be selective and listen to a few to validate them and curate a list. So a fully automated script is probably not the best answer.

    The CSV file can be massaged into a table (e.g. into LibreOffice) with you choosing the final batch.

    [P.S.] I had another thought. Perhaps a professional librarian research tool like Zotero will help you in building collections with attached notes. You need to install a standalone app in your desktop with a browser connector to link to it.

    https://forums.zotero.org/discussion...dcast-citation
    Last edited by dragonfly41; March 17th, 2021 at 10:44 PM. Reason: thought on Zotero

  7. #7
    Join Date
    Nov 2018
    Beans
    12

    Re: Automated Downloading of Radio Show Files

    Thank you for your continued assistance! As I said before, I'm a little out of my depth here. There really isn't much curating involved. The program(me) airs Monday through Friday and each episode comes in three .MP2 files. The shows could include impassioned tributes to evil incarnate or Bulgarian folk music played on an untuned zither and I wouldn't know unless I happened to hear a given day's broadcast (or got an angry telephone call). I download the files for the week ahead and set up the automation playback software, usually on Saturdays. I open the browser, go to the URL, let the browser handle the username and password, and then see a list similar to the one in the picture above, in which each show element is named with a date. The list changes week-to-week as newer shows are added and old ones are pulled off. I make sure all three segments go into their respective folders (Wednesday Segment 1, Friday Segment 3, etc.) and move along. It's a time consuming task that is rather like watching someone ELSE watch paint dry.

    I've been trying to solve this puzzle for months...maybe even years! Many of our other syndicated shows come in via RSS or are distributed via Dropbox, meaning there is little I need to do. I have written Windows batch files (the automation system runs on Win 10) that handle moving the downloads into their proper spots. This show, however, (it really IS a lovely well-produced programme) takes a fair amount of "manual labor."

  8. #8
    Join Date
    Mar 2011
    Location
    U.K.
    Beans
    Hidden!
    Distro
    Ubuntu 20.04 Focal Fossa

    Re: Automated Downloading of Radio Show Files

    Although earlier I listed tools such as Actiona I take the view now that you need a "one stop shop" to manage your workflow.

    Start by installing Krusader (although it comes with KDE baggage).

    Krusader is a dual pane file manager rather like Total Commander in Windows and I suggest that you explore using Krusader desktop as your "mission control centre". The reasons will become clearer after you start using it.

    To install Krusader apply command

    sudo apt install krusader

    You have more control over workflow when using Krusader and in topbar > Settings you can enable

    Show Embedded Terminal
    Show Command Line

    Also we will install glogg as a simple viewer of downloaded CSV file (LibreOffice to open CSV as discussed earlier is overkill).

    sudo apt install glogg

    glogg first column (radiobuttons) allows files to be selected from a large list.

    ===============================================


    Now that you have Krusader installed we will explore a python script for batch downloading MP3 files.

    Searching around I found one script discussion thread close to your needs.

    https://stackoverflow.com/questions/...-using-python3

    Although this discusses downloading Bible readings the list of MP3's is close to your description.

    Start by creating a Project Folder to hold all files.

    Let us say it is in ~/_RADIO

    Create a python script download.py containing this test code (from above thread).

    Code:
    #!/usr/bin/env python
    
    # source
    # https://stackoverflow.com/questions/...-using-python3
    
    
    import requests
    import urllib.request
    import re
    from bs4 import BeautifulSoup
    
    r = requests.get('https://ghalliance.org/resource/bible-reading')
    soup = BeautifulSoup(r.content, 'html.parser')
    
    for a in soup.find_all('a', href=re.compile(r'http.*\.mp3')):
        filename = a['href'][a['href'].rfind("/")+1:]
        doc = requests.get(a['href'])
        with open(filename, 'wb') as f:
            f.write(doc.content)

    Now view this file in Krusader panel and in the embedded terminal

    :~/_RADIO$

    type python download.py

    You will start to see MP3 files being downloaded into the same folder.

    You can see below that I tested to the full and I had to kill the process to stop it running.
    I used Stacer GUI to kill the running python process download.py but you can use Top.
    This workflow can be refined later and filters applied but it is a start.

    I will take a break at this point and explain later how to use Useractions in Krusader.

    These can be used to pipe files into different folders, for example.


  9. #9
    Join Date
    Mar 2011
    Location
    U.K.
    Beans
    Hidden!
    Distro
    Ubuntu 20.04 Focal Fossa

    Re: Automated Downloading of Radio Show Files

    somehow post was duplicated ... removed duplicate
    Attached Images Attached Images
    Last edited by dragonfly41; March 19th, 2021 at 10:37 AM.

  10. #10
    Join Date
    Nov 2018
    Beans
    12

    Re: Automated Downloading of Radio Show Files

    df41: I genuinely appreciate the effort you've made to assist me! I will install Krusader on my test-bed Ubuntu box and play around. As for python...I have no skills whatever in that regard, but I can change things like URLs and directories so we shall see. The biggest challenge (other than having to tackle the task week in and week out) may still be the login for this particular site. We shall see. I'll experiment and report back. Cheers!

Page 1 of 2 12 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •