h264 lossless coding

If x264 does lossless encoding but doesn't like your input format, then your best bet is to use ffmpeg to deal with the input file. Try starting with something like

ffmpeg -i input.avi -f yuv4mpegpipe -pix_fmt yuv420p -y /dev/stdout \
  | x264 $OPTIONS -o output.264 /dev/stdin

and adding options from there. YUV4MPEG is a lossless uncompressed format suitable for piping between different video tools; ffmpeg knows how to write it and x264 knows how to read it.

FFmpeg has a "lossless" mode for x264, see FFmpeg and x264 Encoding Guide

§ Lossless H.264

in essence it's -qp 0

I am going to add a late answer to this one after spending all day trying to figure out how to get YUV 4:4:4 pixels into x264. While x264 does accept raw 4:2:0 pixels in a file, it is really quite difficult getting 4:4:4 pixels passed in. With recent versions of ffmpeg, the following works for completely lossless encoding and extraction to verify the encoding.

First, write your raw yuv 4:4:4 pixels to a file in a planar format. The planes are a set of Y bytes, then the U and V bytes where U and V use 128 as the zero value. Now, invoke ffmpeg and pass in the size of the raw YUV frames as use the "yuv444p" pixel format twice, like so:

ffmpeg -y -s 480x480 -pix_fmt yuv444p -i Tree480.yuv \
-c:v libx264 -pix_fmt yuv444p -profile:v high444 -crf 0 \
-preset:v slow \
Tree480_lossless.m4v

Once the encoding to h264 and wrapping as a Quicktime file is done, one can extract the exact same bytes like so:

ffmpeg -y -i Tree480_lossless.m4v -vcodec rawvideo -pix_fmt yuv444p \
Tree480_m4v_decoded.yuv

Finally, verify the two binary files with diff:

$ diff -s Tree480.yuv Tree480_m4v_decoded.yuv
Files Tree480.yuv and Tree480_m4v_decoded.yuv are identical

Just keep in mind that you need to write the YUV bytes to a file yourself, do not let ffmpeg do any conversion of the YUV values!