Results 1 to 6 of 6

Thread: Make a script for download files

  1. #1
    Join Date
    Jun 2010
    Beans
    151

    Make a script for download files

    Hi all, I've got a question. Is it possible to make a Perl script that automatically would download files from a specific website.
    I was thinking in adding wget commands into that script, but wget command doesn't do what I thought it would do.
    Say the website http://abcabcabc.com/123 allows me to download a file named report.txt. In other words, by putting that URL in the browser the "Save" dialog would show up, and I would be able to download the file.

    I thought the command wget http://abcabcabc/com/123 would download the report.txt. Instead, it download the actual .html file (which shows up with garbage)


    What are my options?

    Thanks.

  2. #2
    Join Date
    Jul 2013
    Location
    Wisconsin
    Beans
    1,690

    Re: Make a script for download files

    Perhaps you misunderstand how web sites and URLs work.

    The scenario you describe only occurs using a web browser to download a link, or if the link is hidden by javascript.
    You want the URL directly to report.txt instead of the 123.html page. For example: http://abcabcabc.com/textfiles/report.txt .

    If report.txt is dynamically generated, then you should manually analyze the page to determine how to get it. For example, by sending the correct POST or Javascript data.

    Using a script to simulate a person traversing the website in a browser, clicking buttons and responding to prompts, is usually a frustrating and ineffective way to go.

  3. #3
    Join Date
    Jun 2010
    Beans
    151

    Re: Make a script for download files

    How would I analyze the website?? I tried typing the website with firebug enabled, but firebug doesn't get to report anything.

  4. #4
    Join Date
    Jul 2013
    Location
    Wisconsin
    Beans
    1,690

    Re: Make a script for download files

    I would read the html and/or javascript code.
    Perhaps it's not dynamically generated. Perhaps you merely have the wrong static URL.

  5. #5
    Join Date
    Jun 2010
    Beans
    151

    Re: Make a script for download files

    Thanks for your reponse.See the weird thing is that the website itself, the one that makes the "Save" dialog to pop up, contains garbage. As I said before I did a wget <website>, and it downloaded the html file, there nothing meaningful..

    Attached is a screen-shot
    Attached Images Attached Images

  6. #6
    Join Date
    Aug 2011
    Location
    47°9′S 126°43W
    Beans
    1,911
    Distro
    Kubuntu 12.10 Quantal Quetzal

    Re: Make a script for download files

    Use an add-on to watch the HTTP headers (like "Live HTTP Headers" for Firefox). What is the content-type of what you see? it could be some Flash thing...

    You can also save one file by hand and check its download URL in the download manager, and see if you can algorithmically deduce the others from it (like continuing a number sequence) or relate that URL to something in the original link, and then algorithmically deduce the URL of the target files from the URLs of the original link in the web page.
    Warning: unless noted otherwise, code in my posts should be understood as "coding suggestions", and its use may require more neurones than the two necessary for Ctrl-C/Ctrl-V.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •