New Internet accountability software: Net Responsibility

[mssever]

I've been working on an Internet accountability program for Linux. The idea is to log the websites you visit, then periodically e-mail that log so someone else. It also writes a log entry when it is shutdown.

I'm looking for people to test it. Any comments (or code contributions) are welcome.

Still to do:

• Make a configuration GUI (partially completed, but on hold for now)
• Increase the intelligence of the shutdown logger
• Build a web service that will provide a friendlier, configurable way for the accountability partner to view the logs

Installation

NOTE: I keep this post up to date, and it is the canonical place for installation instructions, usage, known bugs, etc. The discussion later in the thread often deals with specific issues and/or versions of Net Responsibility, so a given post might not contain current information.

1. Download Net Responsibility. You can get the latest release from the Sourceforge download page. If you're feeling adventurous, you can use Subversion to check out the latest development version:

   svn checkout https://responsibility.svn.sourceforge.net/svnroot/responsibility/trunk net-responsibility

2. If you downloaded the .deb, simply install it. If you downloaded the source tarball, extract it and follow the directions in the README file.
3. Run Applications > Internet > Configure Net Responsibility.

Running

The core of Net Responsibility is a daemon that runs in the background. It is started automatically when you boot your computer. However, if you want to control it manually for testing purposes, here's how:

• Start

  sudo net-responsibility-daemon

• Stop

  sudo /etc/init.d/net-responsibility-daemon stop

Net Accountability sends daily reports via e-mail. Be sure to set the proper e-mail addresses in the config file (/etc/net-responsibility.conf). The easiest way to accomplish this is by running Applications > Internet > Configure Net Responsibility. You can generate a report at any time by typing

sudo net-responsibility-report

Known Bugs
Both net-responsibility-daemon and net-responsibility-report print an error message similar to the following to one or more of stderr, /var/log/syslog, or /var/log/daemon.log:

net-responsibility-report[17580]: The following exception (SQLite3::BusyException) is caused by a known bug. If you know how to fix it, do tell! Otherwise, you can safely ignore it (I hope).
net-responsibility-report[17580]: /usr/lib/ruby/1.8/sqlite3/errors.rb:94:in 'check': Unable to close due to unfinalised statements (SQLite3::BusyException)
net-responsibility-report[17580]: => from /usr/lib/ruby/1.8/sqlite3/database.rb:153:in 'close'
net-responsibility-report[17580]: => from /usr/local/bin/net-responsibility-report:100:in 'initialize'
net-responsibility-report[17580]: => from /usr/local/bin/net-responsibility-report:116:in 'new'
net-responsibility-report[17580]: => from /usr/local/bin/net-responsibility-report:116

(The process name and ID vary, of course.) I haven't been able to solve this error yet, but I don't think that it's serious. Previously, I suppressed it, but I'm not comfortable with suppressing error messages.

[mhancoc7]

Originally Posted by mssever I've been working on an Internet accountability program for Linux. The idea is to log the websites you visit, then periodically e-mail that log so someone else. It also writes a log entry when it is shutdown. I'm looking for people to test it. Any comments (or code contributions) are welcome. Still to do: It doesn't actually send mail yet because I'm not sure yet what the best method to accomplish that is (suggestions are welcome) Make a configuration GUI Make a deb Make the reports more selective, perhaps by using a URL filter; also increase the intelligence of the shutdown logger Installation Install the dependencies: sudo aptitude install ruby dsniff libdaemonize-ruby Download and extract the attached file. Open a terminal, cd to the directory just created, and type sudo ./install.sh Running The core of Net Responsibility is a daemon that runs in the background. It is started automatically when you boot your computer. However, if you want to control it manually for testing purposes, here's how: Start sudo /etc/init.d/net-responsibility-daemon start Stop sudo /etc/init.d/net-responsibility-daemon stop If you want to monitor the log in real time: sudo less /var/log/net-responsibility.log Then press <Shift>F. Net Accountability sends weekly reports. Because it doesn't do real mail yet, the report will be mailed to your machine's root account. You can generate a report at any time by typing sudo net-responsibility-report

This looks very promising. I will be watching this closely.

Thanks, Jereme

[xilix]

OK, now I have tested it. The installation worked and I can check the log file.

All visited websites and their parts are listed (tried Firefox and Opera). This is a lot of stuff and would need very time-consuming reading of the log-file.
Perhaps the next step is to bundle all entries coming from one domain?

[mssever]

Originally Posted by xilix All visited websites and their parts are listed (tried Firefox and Opera).

Yes, this should get most URLs regardless of the browser used (it even catches my weather applet). It doesn't get some Gmail stuff. My theory is that Gmail must use a nonstandard port number for some of its behind-the-scenes work.

This is a lot of stuff and would need very time-consuming reading of the log-file. Perhaps the next step is to bundle all entries coming from one domain?

Did you use the report program?

sudo net-responsibility-report | less

It does that already. Also, I've added in my not-yet-released development version the ability to exclude certain domain names from the reports.

[xilix]

No, I didn't use it. Looks very good. Perhaps later one can make a "tree-view" (like Nautilus-Folder-Tree) in the GUI. Then the accountability partner can choose on which domains he has to look closer...

If I have understood right, the ability to exclude certain domain names is a preference that the person who reads the reports can set. This would mean, that "secure" domains are not viewed in the report. That's a great feature, too.

Do you have already an idea how to transfer the reports to the accountability partner? Do you think it is possible via E-Mail?

[xilix]

I think I know how to start the mailing process periodically. I would suggest using anacron.
It's like cron, but it doesn't assume that the machine is running continuously. It not a real deamon, but it works like one. If your computer is turned off during the intended time to send the mail, this is performed at the next session.
By default anacron can only start root-commands (which is only needed in this case, I think), but I also figured out how to start user-commands with it...

[clmorg01]

I'll download and test it as well. I'm looking forward to this.

[raijinsetsu]

These log files can get pretty extensive. Have you thought about compressing them? or are they already compressed?
Also, could you use a backend such as MySQL, pySQL, etc?
If you used a backend to store the information, it could easily be read from anywhere on the local net.

[clmorg01]

Doing the following command before any url request are in log will give the following error.

sudo net-responsibility-report

chris@chris-desktop:~/Projects/NetResponsiblity_Builds/net-responsibility-0.1$ sudo net-responsibility-report/usr/share/net-responsibility/lib/logfile.rb:169:in `list_dates': undefined method `sort!' for nil:NilClass (NoMethodError) from /usr/share/net-responsibility/lib/logfile.rb:39:in `initialize' from /usr/share/net-responsibility/lib/report.rb:32:in `new' from /usr/share/net-responsibility/lib/report.rb:32:in `initialize' from /usr/local/bin/net-responsibility-report:48:in `new' from /usr/local/bin/net-responsibility-report:48:in `main' from /usr/local/bin/net-responsibility-report:53

After I made some web request, it was fine.

-chris

[mssever]

Originally Posted by xilix No, I didn't use it. Looks very good. Perhaps later one can make a "tree-view" (like Nautilus-Folder-Tree) in the GUI. Then the accountability partner can choose on which domains he has to look closer...

That would be nice, but I don't think that it's possible in an e-mail. Eventually, I might experiment with using Javascript in an HTML mail message, but I suspect that a lot of mail readers disable Javascript.

It might actually be good to create a web service that the accountability partner can log in to have more control over the report. Anyone want to build that?

If I have understood right, the ability to exclude certain domain names is a preference that the person who reads the reports can set. This would mean, that "secure" domains are not viewed in the report. That's a great feature, too.

Actually, it's set on the machine being held accountable. It's not possible for the accountability partner to set it if the report is e-mailed. A web service would be good here, too.

I've thought a bit more about this feature now that you've mentioned it. I'm not sure that it's such a good idea, because it makes it trivial for the person being held accountable to cover his/her tracks. This needs to be re-thought. Should the report tool always generate full reports? Should it have a hard-coded exclude list that only covers stuff that (for example) Firefox downloads automatically? Or should it exclude based on some whitelist accessible from the Internet--assuming that a suitable list exists?

The underlying traffic capturing program, urlsnarf, doesn't grab SSL (secure) URLs. I haven't investigated why, but it likely is due to the encryption. I'm not sure whether this is a feature or a bug.

Originally Posted by xilix I think I know how to start the mailing process periodically. I would suggest using anacron.

Currently, the installer drops a symlink in /etc/cron.weekly, which is (I believe) run by anacron.

Originally Posted by raijinsetsu These log files can get pretty extensive. Have you thought about compressing them? or are they already compressed? Also, could you use a backend such as MySQL, pySQL, etc? If you used a backend to store the information, it could easily be read from anywhere on the local net.

Eventually, I'll probably change the logfile format to SQLite3. MySQL is too much unnecessary overhead. That should take care of compression, as well. My current format has turned out to be tricky to parse, and the report generating code is very fragile. Switching to SQLite3 should help immensely, together with the requisite rewrite of much of the the report code.

But this is a lower priority than getting some other stuff working. Oh, I also plan on including some logfile rotation eventually to ease up on the disk space.

Originally Posted by clmorg01 Doing the following command before any url request are in log will give the following error.

Thanks for the bug report. I believe that it's fixed now. I'll be releasing the next version soon.