Here's the log I get from one of several files I'm having this problem with - exact same files play under VLC or under Windows, but to get smooth 1080 I need the vpau support so...

Setup is 64-bit Ubuntu 12.04 with the latest version of SMplayer installed:

Code:
/usr/bin/mplayer -noquiet -nofs -nomouseinput -vc ffh264vdpau,ffmpeg12vdpau,ffwmv3vdpau,ffvc1vdpau, -sub-fuzziness 1 -identify -msglevel demux=6 -slave -vo vdpau -ao pulse -nokeepaspect -framedrop -nodr -double -input conf=/usr/share/smplayer/input.conf -stop-xscreensaver -wid 56623149 -monitorpixelaspect 1 -*** -embeddedfonts -***-line-spacing 0 -***-font-scale 1 -***-styles /home/ewang/.config/smplayer/styles.*** -fontconfig -font Arial -subfont-autoscale 0 -subfont-osd-scale 20 -subfont-text-scale 20 -subcp ISO-8859-1 -subpos 100 -nocache -osdlevel 0 -slices -channels 2 -af equalizer=0:0:0:0:0:0:0:0:0:0 -softvol -softvol-max 110 /media/USB2_4TB/ForWatching/[Commie] Macross Frontier the Movie ~Sayonara no Tsubasa~ [BD 1080p AAC] [66AE8F11].mkv

MPlayer svn r34540 (Ubuntu), built with gcc-4.6 (C) 2000-2012 MPlayer Team
mplayer: could not connect to socket
mplayer: No such file or directory
Failed to open LIRC support. You will not be able to use your remote control.
Terminal type `unknown' is not defined.

Playing /media/USB2_4TB/ForWatching/[Commie] Macross Frontier the Movie ~Sayonara no Tsubasa~ [BD 1080p AAC] [66AE8F11].mkv.
libavformat version 53.21.0 (external)
Mismatching header version 53.19.0
Configuration: --extra-version='4:0.8.1-0ubuntu1' --arch=amd64 --prefix=/usr --libdir=/usr/lib/x86_64-linux-gnu --enable-vdpau --enable-bzlib --enable-libgsm --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvorbis --enable-pthreads --enable-zlib --enable-libvpx --enable-runtime-cpudetect --enable-libfreetype --enable-vaapi --enable-gpl --enable-postproc --enable-swscale --enable-x11grab --enable-libdc1394 --shlibdir=/usr/lib/x86_64-linux-gnu --enable-shared --disable-static
libavformat file format detected.
[matroska,webm @ 0x7fa60464d940]max_analyze_duration reached
[matroska,webm @ 0x7fa60464d940]Estimating duration from bitrate, this may be inaccurate
ID_VIDEO_ID=0
[lavf] stream 0: video (h264), -vid 0
ID_AUDIO_ID=0
[lavf] stream 1: audio (aac), -aid 0
ID_SUBTITLE_ID=0
[lavf] stream 2: subtitle (***), -sid 0
VIDEO:  [H264]  1916x1076  0bpp  23.976 fps    0.0 kbps ( 0.0 kbyte/s)
Load subtitles in /media/USB2_4TB/ForWatching/
ID_FILENAME=/media/USB2_4TB/ForWatching/[Commie] Macross Frontier the Movie ~Sayonara no Tsubasa~ [BD 1080p AAC] [66AE8F11].mkv
ID_DEMUXER=lavfpref
ID_VIDEO_FORMAT=H264
ID_VIDEO_BITRATE=0
ID_VIDEO_WIDTH=1916
ID_VIDEO_HEIGHT=1076
ID_VIDEO_FPS=23.976
ID_VIDEO_ASPECT=1.7807
ID_AUDIO_FORMAT=MP4A
ID_AUDIO_BITRATE=0
ID_AUDIO_RATE=48000
ID_AUDIO_NCH=6
ID_START_TIME=0.00
ID_LENGTH=6861.63
ID_SEEKABLE=1
ID_CHAPTERS=0
Opening video filter: [*** auto=1]
Couldn't open video filter '***'.
***: cannot add video filter
==========================================================================
Forced video codec: ffh264vdpau
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
libavcodec version 53.35.0 (external)
Mismatching header version 53.32.2
Selected video codec: [ffh264vdpau] vfm: ffmpeg (FFmpeg H.264 (VDPAU))
==========================================================================
ID_VIDEO_CODEC=ffh264vdpau
==========================================================================
Opening audio decoder: [ffmpeg] FFmpeg/libavcodec audio decoders
AUDIO: 48000 Hz, 6 ch, s16le, 0.0 kbit/0.00% (ratio: 0->576000)
ID_AUDIO_BITRATE=0
ID_AUDIO_RATE=48000
ID_AUDIO_NCH=6
Selected audio codec: [ffaac] afm: ffmpeg (FFmpeg AAC (MPEG-2/MPEG-4 Audio))
==========================================================================
AO: [pulse] 48000Hz 2ch floatle (4 bytes per sample)
ID_AUDIO_CODEC=ffaac
Starting playback...


MPlayer interrupted by signal 11 in module: decode video
ID_SIGNAL=11
- MPlayer crashed by bad usage of CPU/FPU/RAM.
  Recompile MPlayer with --enable-debug and make a 'gdb' backtrace and
  disassembly. Details in DOCS/HTML/en/bugreports_what.html#bugreports_crash.
- MPlayer crashed. This shouldn't happen.
  It can be a bug in the MPlayer code _or_ in your drivers _or_ in your
  gcc version. If you think it's MPlayer's fault, please read
  DOCS/HTML/en/bugreports.html and follow the instructions there. We can't and
  won't help unless you provide this information when reporting a possible bug.