Page 2 of 12 FirstFirst 1234 ... LastLast
Results 11 to 20 of 111

Thread: download torrents from rss feed

  1. #11
    Join Date
    Sep 2009
    Beans
    76
    Distro
    Ubuntu 9.04 Jaunty Jackalope

    Re: download torrents from rss feed

    Is it possible for you to share the feed URL with others ( it would be easier to understand, what exactly you need ) ?

  2. #12
    Join Date
    Apr 2006
    Location
    Coventry
    Beans
    1,379
    Distro
    Ubuntu 9.10 Karmic Koala

  3. #13
    Join Date
    Apr 2006
    Location
    Coventry
    Beans
    1,379
    Distro
    Ubuntu 9.10 Karmic Koala

  4. #14
    Join Date
    Apr 2006
    Location
    Coventry
    Beans
    1,379
    Distro
    Ubuntu 9.10 Karmic Koala

    Re: download torrents from rss feed

    I think I have figured out the terminology. What I am looking to do is essentially "Broadcatching" (http://en.wikipedia.org/wiki/Broadcatching).

    There is a lot of information on it on the web, but other than the Java app TED there does not seem to be anything particular useful for Ubuntu.

    Ideally, I'd want a script that runs every 10-15 minutes and searches the specified feed for any new torrent files, and if there are any new torrent files it should download it to a folder which is then being scanned by Transmission (creating an autodownload function).

  5. #15
    Join Date
    Mar 2008
    Beans
    4,715
    Distro
    Ubuntu 9.10 Karmic Koala

    Re: download torrents from rss feed

    abhiroopb, are you looking for a script to monitor a url web page, check for new torrents (differently named files?) and then issue a command based on the new files found?
    If that is all it takes, I think I can write a script to do that.

    Is this the page you want monitored:
    http://showrss.karmorra.info/rss.php...&hd=0&proper=0

    and what is the command you'd like issued based on the new feeds found?

  6. #16
    Join Date
    Apr 2006
    Location
    Coventry
    Beans
    1,379
    Distro
    Ubuntu 9.10 Karmic Koala

    Re: download torrents from rss feed

    Thats essentially it, unutbu.

    Basically, I have configured that particular website to create a personalised RSS feed for me to only provide me with the torrents I require. So, every single torrent that shows up in that feed would have to be downloaded to my specially assigned "torrents" folder that is being watched by Tranmission.

    It seems simple enough, but I have no idea how to go about it.

    Thanks a lot for your help.

  7. #17
    Join Date
    Mar 2008
    Beans
    4,715
    Distro
    Ubuntu 9.10 Karmic Koala

    Re: download torrents from rss feed

    Okay, once I have the list of new torrents, such as

    http://torrent.zoink.it/True.Blood.S...ztv%5D.torrent

    What does one do with this? Do you download this file to your computer and put it in a certain directory? Which directory? (As this makes abundantly clear, I am a torrent noob...)

  8. #18
    Join Date
    Apr 2006
    Location
    Coventry
    Beans
    1,379
    Distro
    Ubuntu 9.10 Karmic Koala

    Re: download torrents from rss feed

    Thats pretty much all you have to do, once it appears in the specific directory, as long as your torrent client is running it should automatically add it.

    All I would need the script to do is this:

    1. Search RSS feed every few minutes
    2. Download NEW .torrent file (this is obviously important as I don't want the same .torrent file being downloaded over and over again).
    3. Place newly downloaded .torrent file in specified directory.

    Then transmission (my bittorrent client of choice would download the torrent after scanning the specified directory.

  9. #19
    Join Date
    Mar 2008
    Beans
    4,715
    Distro
    Ubuntu 9.10 Karmic Koala

    Re: download torrents from rss feed

    I know you do not need such explicit directions abhiroopb, but I'm writing it this way just in case others find this useful...

    • Install the python-beautifulsoup package
    • Save this to ~/bin/tormon.py

      PHP Code:
      #!/usr/bin/env python

      import urllib2,urlparse
      from urllib2 import HTTPError
      ,URLError
      from BeautifulSoup import BeautifulSoup
      import os
      import optparse

      __usage__
      ='''
      tormon.py -O ~/test/tormon -u "http://rss.feed"
      '''

      class Main(object):
          
      '''
          tormon checks an rss feed for new torrents. When it finds a new .torrent, to
          downloads it to a specified output directory, where (presumably) a monitoring
          torrent program will download the corresponding file.    
          '''
          
      def parse_options(self):
              
      usage 'usage: %prog [options]'+__usage__
              parser 
      optparse.OptionParser(usage=usage)
              
      parser.add_option(
                  
      '-O''--output_dir'dest='output_dir'
                  
      help='directory into which new torrents are saved'
                  
      metavar='DIR')
              
      parser.add_option(
                  
      '-f''--filetype'dest='filetype',
                  
      action='append',
                  default=[],
                  
      help='admissible file types'
                  
      metavar='TYPE')
              
      parser.add_option(
                  
      '-d''--downloaded_torrents'dest='downloaded_torrents',
                  default=
      os.path.expanduser('~/.downloaded_torrents'),
                  
      help='log of already downloaded torrents'
                  
      metavar='FILE')
              
      parser.add_option(
                  
      '-e''--error_log'dest='error_log',
                  
      help='log of torrents tormon failed to download'
                  
      metavar='FILE')
              
      parser.add_option(
                  
      '-b''--batch'dest='batch',
                  
      help='file containing list of rss feed urls'
                  
      metavar='FILE'
              
      parser.add_option(
                  
      '-u''--url'dest='url',
                  
      action='append',
                  default=[],
                  
      help='url of the rss feed'
                  
      metavar='URL')
              
      parser.add_option(
                  
      '-m','--mark_all_downloaded'dest='mark_all_downloaded',
                  
      action='store_true'
                  default=
      False,
                  
      help="mark all torrents as already downloaded")
              
      parser.add_option(
                  
      '-M','--match_by_filename'dest='match_by_filename',
                  
      action='store_true'
                  default=
      False,
                  
      help="recognize downloaded files by filename, not URL. Matching by URL is the default.")        
              (
      self.optargs) = parser.parse_args()
              if 
      self.opt.batch:
                  for 
      line in open(self.opt.batch,'r'):
                      
      line=line.strip()
                      if 
      line and not line.startswith('#'):
                          
      self.opt.url.append(line)
              if 
      not self.opt.output_dir:
                  
      self.opt.output_dir=os.path.expanduser('~/Desktop')
              if 
      not self.opt.filetype:
                  
      self.opt.filetype=['.torrent']
              if 
      not self.opt.error_log:
                  
      self.opt.error_log=self.opt.downloaded_torrents+'.errors'
              
      try:
                  
      os.makedirs(self.opt.output_dir)
              
      except OSError:
                  if 
      not os.path.exists(self.opt.output_dir):
                      print(
      'tormon failed to create directory %s'%self.opt.output_dir)
                      exit(
      1)
          
      def load_list_of_already_downloaded_torrents(self):
              try:
                  
      self.downloaded=open(self.opt.downloaded_torrents,'r').read().split()
              
      except IOError:
                  
      self.downloaded=[]
              try:
                  
      self.errors=open(self.opt.error_log,'r').read().split()
              
      except IOError:
                  
      self.errors=[]
          
      def update_downloaded(self,url):
              
      self.downloaded.append(url)
              try:
                  
      self.errors.remove(url)
              
      except ValueError:
                  
      pass        
          def download_torrent
      (self,url):
              try:
                  
      sock=urllib2.urlopen(url)
              
      except (HTTPErrorURLError):
                  
      # print('tormon failed to download %s'%url)
                  
      if url not in self.errors:
                      
      self.errors.append(url)
              else:
                  
      filename=self.url2filename(url)
                  
      target_file=os.path.join(self.opt.output_dir,filename)
                  print(
      'Downloading %s'%target_file)
                  
      content=sock.read()
                  
      sock.close()
                  
      fh=open(target_file,'w')
                  
      fh.write(content)
                  
      fh.close()
                  
      self.update_downloaded(url)
          
      def url2filename(self,url):
              return 
      os.path.basename(urlparse.urlparse(url)[2])
          
      def has_been_downloaded(self,url):
              if 
      self.opt.match_by_filename:
                  
      filename=self.url2filename(url)
                  return (
      filename in [self.url2filename(link) for link in self.downloaded])
              else:
                  return (
      url in self.downloaded)
          
      def parse_rss_feed(self):
              for 
      url in self.opt.url:
                  print(
      'RSS feed: %s'%url)
                  try:
                      
      sock=urllib2.urlopen(url)
                  
      except (HTTPErrorURLError):
                      print(
      'tormon failed to download %s'%url)
                  else:
                      
      content=sock.read()
                      
      sock.close()
                      
      soup=BeautifulSoup(content)
                      
      links=([link.nextSibling for link in soup.findAll('link')]+
                             [
      link['href'] for link in soup.findAll('a')]+
                             [
      link['url'] for link in soup.findAll('media:content')])
                      for 
      link in links:
                          if (
      any([link.lower().endswith(ending)
                                   for 
      ending in self.opt.filetype])
                              and 
      not self.has_been_downloaded(link)):
                              if 
      self.opt.mark_all_downloaded:
                                  print(
      'Marking %s as downloaded'%link)
                                  
      self.update_downloaded(link)
                              else:
                                  
      self.download_torrent(link)
          
      def save_list_of_already_downloaded_torrents(self):
              
      fh=open(self.opt.downloaded_torrents'w')
              
      fh.write('\n'.join(self.downloaded))
              
      fh.close()
              
      fh=open(self.opt.error_log'w')
              
      fh.write('\n'.join(self.errors))
              
      fh.close()
          
      def __init__(self):
              
      self.parse_options()        
              
      self.load_list_of_already_downloaded_torrents()
              try:
                  
      self.parse_rss_feed()
              
      except KeyboardInterrupt:
                  
      pass
              finally
      :
                  
      self.save_list_of_already_downloaded_torrents()
      if 
      __name__=='__main__':
          
      Main() 
    • Make it executable:
      Code:
      chmod +x ~/bin/tormon.py
    • Test it:
      Code:
      tormon.py -O /path/to/downloaded/dot/torrent/files -u "http://rss.feed"
      Putting the url of the rss feed in quotes protects the url from being misinterpreted by the shell.
      • /path/to/downloaded/dot/torrent/files should be the directory where you wish the .torrents to be saved.
      • If you omit the -O flag, the files will be placed in ~/Desktop.
      • You should see a list of the torrents downloaded.
      • If you run the same command again, nothing should be downloaded unless a new torrent has been added to the rss feed.
      • This command creates /path/to/downloaded/dot/torrent/files if it does not already exist.
      • It will save a list of the urls of successfully downloaded torrents to ~/.downloaded_torrents.
      • I tried to make the script general enough to be of use to other people too. If some else has a rss feed they wish to monitor, they can use the command
        Code:
        tormon.py -u "http://rss.feed"
      • To monitor more than one rss feed, you can use the -u flag multiple times:
        Code:
        tormon.py -u "http://rss.feed1" -u "http://rss.feed2"
      • ~/.downloaded_torrents is a plain text file. If you wish to download the same torrent again, just delete the url from this file. If you wish to "blacklist" a certain torrent, you can just add the url to this file.
      • If you want to save the list of downloaded torrents to a different file, you can use the command
        Code:
        tormon.py -d /path/to/downloaded/torrents/file -u "http://rss.feed1"
    • If everything seems to be working properly, then set up the cron job: Run
      Code:
      crontab -e
      A text editor (e.g. nano) will pop up.

      Add this to the top of the file, changing USER to your username (e.g. abhiroopb)
      Code:
      PATH=/sbin:/bin:/usr/sbin:/usr/bin:/home/USER/bin
      MAILTO=USER
      Put this anywhere under the "PATH=..." line.
      Code:
      */15 * * * * tormon.py -u "http://rss.feed"
      This will run the tormon.py command once every fifteen minutes.

      Make sure there is at least one blank line at the end of the crontab file. (It's a quirk of the cron program.)
      Save and exit the text editor.

      If you have a mail transfer agent (e.g. exim4) installed, then the MAILTO=USER line in the crontab will tell cron to send you the output of the command in a (local) email message.
      You then get to see what torrents have been most recently downloaded.
    • That's it. See if it works. I'll be expecting your bug reports. Enjoy!
    Last edited by unutbu; November 22nd, 2009 at 04:48 AM.

  10. #20
    Join Date
    Apr 2006
    Location
    Coventry
    Beans
    1,379
    Distro
    Ubuntu 9.10 Karmic Koala

    Re: download torrents from rss feed

    Hi there.

    WOW! That is amazing! Its perfect, and it means I don't have to be running a clunky java programme in the background all day long.

    I really love what you can do with linux!

    Thanks again, like I said this is exactly what I was looking for, and with the help of the linux community it took me less than a day to find it!

    NB: I have PMed you about something else.

Page 2 of 12 FirstFirst 1234 ... LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •