Page 2 of 2 FirstFirst 12
Results 11 to 14 of 14

Thread: Automated Downloading of Radio Show Files

  1. #11
    Join Date
    Dec 2014
    Beans
    1,741

    Re: Automated Downloading of Radio Show Files

    If you read the manual page for wget ('man wget' in a shell), you'll find that there are ways to simulate a login to a modern page that uses scripting both on the server and in the browser to do the login page. The '--user' and '--password' options of wget are for the old method of user authentication which is built into the server and is used only rarely today mostly for aesthetic reason (the form for this method is build into the browser and is rather ugly and can't be styled in any way; also it's as insecure as can be - username and password are transmitted in the clear as part of the URL).

    For newer login methods, you'd have to look at the html-source of the login page, find the URL to which the data is sent and the names used for the fields. Then call wget with the options --save-cookies (with a filename to save the cookies in) and --post-data with the names and values for the fields in the login form with the url of the script that validates the login. You will probably also need the option --keep-session-cookies if the pages uses session cookies (cookies that are normally not meant to be stored and should be forgotten as soon as the browser is closed). After that you should be able to grab the files with wget by using the option --load-cookies with the file into which you saved the cookies.

    Holger
    Last edited by Holger_Gehrke; March 19th, 2021 at 12:54 AM.

  2. #12
    Join Date
    Mar 2011
    Location
    U.K.
    Beans
    Hidden!
    Distro
    Ubuntu 20.04 Focal Fossa

    Re: Automated Downloading of Radio Show Files

    There are several methods to access browser elements (such as login fields) through cookies and selenium.

    But the following workflow might be easier.
    It draws on the bot utility Actiona which I cited earlier.

    Code:
    sudo apt install actiona
    We will prepare a directory of sites where each login profile requires these variables:


    • url of main site for logon
    • url of download site where mp3's are to be downloaded after login
    • x:y location of user field
    • x:y location of password field
    • user field: username
    • password field: password
    • path where mp3 files are to be downloaded after login


    This information can be gathered together in one setup (calibration) session. After the datatable is gathered the process will be automatic. There can be as many sites as you wish.

    The main site and download site might have the same url. In this case just duplicate the url in the dataset.

    This directory will be embedded in Actiona script as a Code object.

    Actiona does have a Code object editor but the fonts are small and it might be easier to write long lists in an external editor then paste into Actiona Code object editor.

    An editor I use is Synaptic Sublime 3 . Importantly use Tools > Package Control: Install Package, and install Actionscript package for syntax highlighting. But you can use any editor which suits your workflow.

    Now a multi item variable will look like this:

    Code:
    var sites =  [
    ["https://mainsite1", "https://downloadsite1", "x:y", "x:y", "username1",  "password1", "path1"],
    ["https://mainsite2", "https://downloadsite2", "x:y", "x:y", "username2",  "password2", "path2"]
    ];
    To capture the positions of each target login field, we can use a script as explained here.

    https://askubuntu.com/questions/5378...e-mouse-cursor

    Run this command

    Code:
    while true; do clear; xdotool getmouselocation; sleep 0.1; done
    And next maximise browser (so that coordinates are consistent) and navigate to each site requiring login parameters.

    As you move the cursor to be positioned in login fields, write down the monitored coordinates x:y.

    These coordinates will be written into the datatable.

    The target sites are selected from a MultiData Object using notation:

    Code:
    $sites[0][1]
    .. this picks up first x:y

    Now when Actiona runs we can invoke the Move Cursor object to position cursor.

    Username will be picked up and pasted into the username field.
    Click object is run.

    We repeat this with password field.

    This takes us into the site and we can run a Command object to run a python script to download mp3 files and place in designated folder.

    I will follow up later with a sample Actiona script (but I do not have multiple site details so you will have to configure yourself).

    I use this approach when navigating cloud sites such as AWS console, Google console.
    Last edited by dragonfly41; March 19th, 2021 at 11:20 PM. Reason: corrected code syntax var sites = [};

  3. #13
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: Automated Downloading of Radio Show Files

    cnee can record and playback steps taken.
    Code:
    # to get the location of mouse clicks
    cnee --record --mouse | awk  '/7,4,0,0,1/ { system("xdotool getmouselocation") }'
    xdotool can be used to enter text, move the mouse, click, send events to specific windows, launch other programs. The main problem is that all mouse and data entry is positional when we really want to use the widgetID instead.

    To record a cnee session, use:
    Code:
    # Record
    cnee -t 2 -rec --first-last  --keyboard --mouse -o test.xms  -sk q
    To replay that session:
    Code:
    # Replay
    cnee -rep --synchronised-replay  -f  test.xms
    Quick and dirty, but sometimes that's all we need. If you can launch all programs to have specific geometries and placement in a workspace, then success is nearly assured. Nearly.

    As usual, there are 50 ways to solve any problem. cnee being able to record and playback is HUGE!

  4. #14
    Join Date
    Mar 2011
    Location
    U.K.
    Beans
    Hidden!
    Distro
    Ubuntu 20.04 Focal Fossa

    Re: Automated Downloading of Radio Show Files

    I did not know about cnee but I have used xdotool (see earlier post #12 where it is used).
    In fact it can be run through Actiona Command as one of the created actions in the GUI.
    Window Condition.

    I saw error #24 when running cnee ..

    https://savannah.gnu.org/bugs/?44914

    we really want to use the widgetID instead

    Note that Actiona does exactly that and more.

    But these are all horses for courses. Multiple approaches.
    To understand Actiona coding it pays to read through Actionscript2: The Definitive Guide by author Colin Moock going back to Flash days some years ago.
    Last edited by dragonfly41; March 20th, 2021 at 04:40 PM.

Page 2 of 2 FirstFirst 12

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •