I'm not very familiar with Myth, but in order to reproduce 1080p video you need to:
- Have the proper driver installed. In this case the Nvidia proprietary driver.
- Have the proper libraries installed: libvdpau1.
- Make sure the video is encoded in one of the supported formats (basically h264, x264 or MPEG).
- Make sure that the player is both able to use VDPAU and is set to use it as an output (instead of XVideo and X11).
In order to see if the library is install, you can run this:
Check the proper Myth documentation for the output setting.
apt-cache policy libvdpau1
Hope it helps.