zmatt-
Saw your post on another forum as well. Are you still stuck with video issues?

That first picture looks similar to an issue I've seen in other desktops. It looks almost there but maybe EXA or composite issues. (workarounds to those in FAQ/Known Issues)
The second "psychedelic" is expected with modeset 0.

Did you try ruling out acceleration?
Code:
boot: live nouveau.noaccel=1
If you can CTL-ALT-F1 or CTL-ALT-T to a prompt and get a copy of /var/log/Xorg.0.log and post it, others might be able to help more.

Also, IMHO, it is good practice to look up and post your video card model when discussing graphics issues so readers know exactly what hardware is being discussed. I'm assuming yours has Nvidia Go5200. everymac.com is a good source to look up.