Mkv is a container, x264 is a video codec. The mkv file could well contain x264 video, but it's the audio you're interested in, not the video, unless the reason why the audio is jumping is because the ps3 is having trouble decoding the video.
Is the video HD or very high bitrate?
My guess is that the mkv file contains audio encoded with a dodgy codec that the ps3 doesn't like. You could try extracting the video and audio streams and re-encoding the audio.
Maybe the ps3 doesn't like MKV, you could try ripping the streams and making an mp4 file.
Sorry, never owned a ps3 so this is all guess work :S