Short and to the point: You need to read up on the various MPEG specs and associated MPEG Levels. MPEG-2 (DVD) is different form MPEG-2 for HD TV/ BluRay and so on. Same for H.264. There is nothing wrong with AE you just need to make an effort to actually try and educate yourself here and based on that use the correct settings, even more so since in CS5 the program does not automatically match settings to the comp...
Mylenium