How do I make it so that a single app could not kill my entire system just by taking all the RAM?

Often times a rogue app starts to chew up all my RAM and by the time I realize it, my hard drive is swapping faster than a swingers couple and several seconds later even my cursor is not responding.

I found this page, but it talks about limiting particular apps:

However, how to do this globally, so that any app that behaves in a dangerous way would be killed?

I believe something like this was a default in the days of Edgy, as my IRC bot would often get killed by the system if it ran too long (it had a memory leak). What happened to that? Another move closer to being Windows?