Command line RSS reader for TVrss wanted

[neilp85]

Does anyone know of a decent command line RSS reader that I could use for downloading torrents from TVrss feeds. I've been doing a decent amount of searching on the net for something but haven't been able to find anything to suit my needs. I could probably write something myself to solve this problem but would prefer not to.

[neilp85]

I hacked out a solution to this a while back so I figured I would post it for everyone. It's pretty ugly (one of my first python scripts), but does exactly what I need. There are a couple things I planned on fixing but never got around to. You will need to install the universal feed parser package for this to work.

#!/usr/bin/env python
"""torrentdl.py - Downloads new torrents from tvrss.net feeds"""

import feedparser, os, time

path = os.getenv('HOME') + '/downloads/torrents'

# Load every line of the file to a list element
file = open(path + '/tvrss-feeds')
lines = [line for line in file.readlines() if line.strip()]
file.close()

# Last line of the file should be the previous time we
# checked the feeds, if not make it the current time.
if lines[-1][0] == '(':
    # Force it to the proper format
    prev_time = lines[-1].strip()[1:-1]
    prev_time = (prev_time.split(', '))
    prev_time = tuple([int(val) for val in prev_time])
    lines.pop()
else:
    prev_time = time.gmtime()

# Generate a list of rss urls by removing comments
url_list = [url.strip() for url in lines if url[0] != '#']

# Get the current time (GMT because feedparser uses it) to
# replace old time value at bottom of file
curr_time = time.gmtime()

# TODO: This could really be sped up if each feed was handled
#            in it's own thread from here on

# Grab all the rss feeds
feeds = [feedparser.parse(rss_url.strip()) for rss_url in url_list]

# Basic wget command for all torrent downloads
# --referer is a workaround for torrents from mininova
base_command = 'wget --quiet --tries=1 --read-timeout=90 --referer=%s %s'

# Make sure torrents get dowloaded to proper directory
os.chdir(path)

# Check all your feeds for new torrents
for feed in feeds:
    for entry in feed['entries']:
        # Only download new torrents
        if(prev_time < entry['updated_parsed']):
            # Check that the torrent was actually downloaded
            command = base_command % (entry['link'], entry['link'])
            print entry['summary']
            if(os.system(command) != 0):
                # TODO: Error handling
                pass

# Overwrite the old file with the new time value
file = open(path + '/tvrss-feeds', 'w')
file.writelines(lines)
curr_time = [str(val) for val in curr_time]
file.write("(" + ", ".join(curr_time) + ")" + '\n')
file.close()

Find the shows you want on tvrss then add the RSS links to your feeds file and off you go. I use this as a cron job in conjunction with rtorrent watching the directory torrents are downloaded to. Feel free to critique and post any improvements.

[Mateo]

does this have to be used with torrents? could it work with podcasts or something like that too?

I've been looking for a solution similar to this to use in conjuction with youtube-dl, this might work. i'll take a look at it later more closely.

[neilp85]

This could probably be modified to be a more generic RSS content downloader. You would need an abstraction for obtaining the content link and action to perform on it. I'm gonna take a look at it and see what I come up with.

[neilp85]

Alright, after some work I've come up with a more generic version. I also cleaned up the code some. Besides adding the feed to the file with all your others, the only thing you should have to do is provide an action for new types of feeds (i.e. youtube, tvrss, etc...). I added a basic one for youtube videos using youtube-dl. There's still some more modifications I want to make, but this is a pretty good start.

#!/usr/bin/env python
"""rssdl.py - Performs acctions on new content linked to from rss feeds"""

import feedparser, os, time

# Retrieves value from dictionary with partial match key
def getValue(key, dict):
    for k in dict.keys():
        if key.startswith(k):
            return actions[k]
    return None


# Actions are dependent on the type of feed
actions = { \
    'tvRSS': 'wget --quiet --tries=1 --read-timeout=90 %(link)s', \
    'YouTube': 'youtube-dl --output="%(title)s.flv" %(link)s', \
}

path = '%s/downloads' % os.getenv('HOME')
name = '%s/feeds' % path

# Load every line of the file to a list element
file = open(name)
lines = [line for line in file.readlines() if line.strip()]
file.close()

# Last line of the file should be the previous time we
# checked the feeds, if not make it the current time.
if lines[-1][0] == '(':
    # Force it to the proper format
    oldtime = lines[-1].strip()[1:-1]
    oldtime = (oldtime.split(', '))
    oldtime = tuple([int(val) for val in oldtime])
    lines.pop()
else:
    oldtime = time.gmtime()

# Generate a list of rss urls by removing comments
urls = [line.strip() for line in lines if line[0] != '#']

# Get the current time (GMT because feedparser uses it) to
# replace old time value at bottom of file
newtime = time.gmtime()

#TODO: This could really be sped up if each feed was handled
#      in it's own thread from here on

# Grab all the rss feeds
feeds = [feedparser.parse(url.strip()) for url in urls]

# Make sure torrents get dowloaded to proper directory
os.chdir(path)

# Check all your feeds for new torrents
for feed in feeds:

    # Ignore feeds we don't know what to do with
    action = getValue(feed['feed']['title'], actions)
    if action is None:
        continue

    for entry in feed['entries']:
    # Only download new torrents
        if oldtime < entry['updated_parsed']:
            # Check that action was taken
            if os.system(action % entry) != 0:
                # TODO: Error handling
                pass


# Overwrite the old file with the new time value
file = open(name, 'w')
file.writelines(lines)
newtime = [str(val) for val in newtime]
file.write("(%s)\n" % ", ".join(newtime))
file.close()

# End of file rssdl.py

[neilp85]

Could a moderator move this thread to programming talk. It seems like a much more appropriate place for it now.

[Mateo]

ohh wow, I wasn't expecting you to do all of that. thanks a lot, i'll take a look at it and see if it needs any tweaking.

[tbroderick]

You can use raggle or snownews with a console web browser like lynx or elinks. Add the feed and open the entry with the browser, it will ask you to save the .torrent.

[neilp85]

The problem with those is that they won't automatically download the torrent file when a new entry is added to the RSS feed. I would have to check them periodically and go download any files. However, my solution is fully automated so the torrents will start downloading as soon as they are available.

[casaschi]

It uses a different RSS feed, but you can automate torrent downloads from RSS using tvtorrentfetch ( http://xyzabc.xy.ohost.de/ )