I finally learned how to properly manage switching back and forth between priprietary and open-source drivers for my ATI Radeon HD 5750 (somehow, sometimes NVidia graphic drivers get installed and then the system can't load GLX modules), so I embarked on a quest to get hardware accelerated video playback on my machine since I decided to keep fglrx. Since I mostly use UMplayer for video playback, I found out I had to compile vaapi-enabled branch of it (from http://gitorious.org/vaapi/mplayer). Now, I did that and hardware accelerated playback seems to work.
CPU usage drops from about 50% to a mere 10% when I add -vo vaapi to the mplayer command to play a 720p video, but no matter what I do I can't get it to work in UMplayer nor SMplayer. I tried setting vaapi for Video output driver and adding -vo vaapi to the "Options for MPlayer", but all I get is a UMplayer logo instead of the movie being played.
How do I make videos play inside U/SMplayer?
UPDATE: Starting video with $ mplayer file.mp4-vo vaapi:gl -va vaapi seems to pause it at first frame. Maybe that's the problem?
Also, this is the output:
UPDATE2: Okay, so I messed up compiling, two different versions were at /usr/bin and /usr/local/bin; players picked the latter while bash would choose the former. That is fixed now and I can play video in SMplayer (not in UMplayer - X11 Error (BadMatch) occurs whether I select vaapi or any other output driver). So i played the same movie in SMplayer and through command line. They use the same -vo settings. The SMplayer is still using 50% CPU while the command line one uses 9. This sucks.
MPlayer SVN-r35107-4.6 (C) 2000-2012 MPlayer Team
libavformat version 54.25.104 (internal)
libavformat file format detected.
[lavf] stream 0: video (h264), -vid 0
[lavf] stream 1: audio (ac3), -aid 0, -alang und
[lavf] stream 2: audio (aac), -aid 1, -alang und
VIDEO: [H264] 1280x544 24bpp 23.976 fps 2619.1 kbps (319.7 kbyte/s)
creation_time: 1940-08-15 17:43:07
Load subtitles in ./
[vo_vaapi] Using OpenGL rendering
libva: VA-API version 0.32.0
Xlib: extension "XFree86-DRI" missing on display ":0".
libva: va_getDriverName() returns 0
libva: Trying to open /usr/lib/i386-linux-gnu/dri/fglrx_drv_video.so
libva: va_openDriver() returns 0
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
libavcodec version 54.53.100 (internal)
Selected video codec: [ffh264] vfm: ffmpeg (FFmpeg H.264)
Opening audio decoder: [ffmpeg] FFmpeg/libavcodec audio decoders
AUDIO: 48000 Hz, 2 ch, s16le, 384.0 kbit/25.00% (ratio: 48000->192000)
Selected audio codec: [ffac3] afm: ffmpeg (FFmpeg AC-3)
[AO OSS] audio_setup: Can't open audio device /dev/dsp: No such file or directory
AO: [alsa] 48000Hz 2ch s16le (2 bytes per sample)
Unsupported PixelFormat 61
[VD_FFMPEG] Trying pixfmt=1.
Movie-Aspect is 2.35:1 - prescaling to correct movie aspect.
VO: [vaapi] 1280x544 => 1280x544 H.264 VA-API Acceleration
[VD_FFMPEG] XVMC-accelerated MPEG-2.
A: 0.5 V: 0.5 A-V: -0.080 ct: -0.029 0/ 0 140% 16% 0.8% 8 0