How to set the MediaRecorder to get the best video quality effect?

Depending on the API level you may want to use existing profiles or not.

Without profiles:

recorder.setVideoSize(640, 480);
recorder.setVideoFrameRate(16); //might be auto-determined due to lighting
recorder.setVideoEncodingBitRate(3000000);
recorder.setVideoEncoder(MediaRecorder.VideoEncoder.H264);// MPEG_4_SP
recorder.setAudioEncoder(MediaRecorder.AudioEncoder.AMR_NB);

Or if you want to use existing profiles

CamcorderProfile cpHigh = CamcorderProfile.get(CamcorderProfile.QUALITY_HIGH);
recorder.setProfile(cpHigh);

Please note that you cannot have both options together as you will get errors or your prepare will not work

As not all the Android API and/or devices support the same values you will either have to query the maximum values per device or find something that works everywhere.

Although the question is quite old, i would like to to point out the combination i used to record the video of HD quality.

Use below combination of the code to achieve HD quality video.

CamcorderProfile cpHigh = CamcorderProfile.get(CamcorderProfile.QUALITY_HIGH);                    
mMediaRecorder.setAudioSource(MediaRecorder.AudioSource.MIC);                        
mMediaRecorder.setVideoSource(MediaRecorder.VideoSource.SURFACE);                        
mMediaRecorder.setOutputFormat(MediaRecorder.OutputFormat.THREE_GPP);
mMediaRecorder.setVideoSize(DISPLAY_WIDTH, DISPLAY_HEIGHT);                    
mMediaRecorder.setVideoEncoder(MediaRecorder.VideoEncoder.H264);                    
mMediaRecorder.setAudioEncoder(MediaRecorder.AudioEncoder.AMR_NB);                    
mMediaRecorder.setVideoEncodingBitRate(cpHigh.videoBitRate);                    
mMediaRecorder.setVideoFrameRate(cpHigh.videoFrameRate);
int rotation = mWindowManager.getDefaultDisplay().getRotation();
int orientation = ORIENTATIONS.get(rotation + 90);
mMediaRecorder.setOrientationHint(orientation);

Use below code to get the DISPLAY_HEIGHT,DISPLAY_WIDTH

DisplayMetrics metrics = new DisplayMetrics();      
mWindowManager.getDefaultDisplay().getMetrics(metrics);          
DISPLAY_WIDTH = metrics.widthPixels;
DISPLAY_HEIGHT = metrics.heightPixels;

Define ORIENTATIONS as shown below

public static final SparseIntArray ORIENTATIONS = new SparseIntArray();
static {
    ORIENTATIONS.append(Surface.ROTATION_0, 90);
    ORIENTATIONS.append(Surface.ROTATION_90, 0);
    ORIENTATIONS.append(Surface.ROTATION_180, 270);
    ORIENTATIONS.append(Surface.ROTATION_270, 180);
}

Hope it helps.