This is odd. I do know that by default stdout is line buffered, so if mencoder uses some kind of fancy output scheme (like ffmpeg does) then this might be messing with the buffering scheme on stdout which is in turn messing with Python. As from what I can tell running tests with various other UNIX programs, there doesn't seem to be a limit of output lines that can be collected by doing:
To solve the problem, there is probably an option for turning on script friendly output with mencoder. Have you checked already?
>>> import subprocess
>>> p = subprocess.Popen(['command'], stdout=subprocess.PIPE)
>>> while p.returncode is None:
... print p.stdout.readline()