yes it sounds like a compat pb.
With x264 from svn, it's ok (adding a line of code in libavcodec/nellymoserenc.c).
About graphic, i don't know what went wrong. try after try i could configure with --enable-gui, and could compile ;
the only way the alter XShape cap is from xorg.conf, and obviously i tried with native xorg nvidia driver and with the proprietary one... Just using the admin menu... a non deterministic automated process
Without --enable-gui, everything is ok.