video encoding software - multi process

bigredshark

bigredshark

bigredshark

Man of Honour
Joined
30 Jun 2005
Posts
9,515
Location
London Town!
A quick one...

I want to encode some video under linux and I currently use handbrake or mencoder, which works for me but I'd like a little extra performance.

I'm considering setting up a OpenMosix cluster to improve the performance. This spreads load by migrating processes between machines, however handbrake only appears to spawn multiple threads rather than processes.

So, can anybody recommend encoding software which will spawn multiple processes?
 
This is a very interesting prospect. Using a render farm for rendering 3D video is a long-established practice but I'm not familiar with any good solutions for encoding/reincoding existing video.

A quick session with Google suggests that MEncoder might be able to do this for all formats supported by Mplayer, which would be neat-o. I'm looking more into it, as should you.
 
Yes, I suspected mencoder would be the what I needed in the end to do exactly that. It brings me to a bit of a decision though...

My original idea was to spread the encoding of a single file between multiple machines (tricky but should be doable providing I can get something to spawn multiple processes and store the input and output file on a fast NAS.) Then I'd simply encode files one after another (which suits me better as I may want to do one file in a hurry)

I can encode a full DVD (ripped to an ISO file and mounted first) in the quality settings I need in about 15 minutes. (I tested on a 4x 3Ghz Xeon box at work which I wasn't doing anything else with). Given this kind of performance from one big box, the need to distribute the encoding of a single file reduces and I could just encode different files in parallel on each machine.

I'm going to keep looking at it (mainly I have to say because it'd be pretty cool to encode an entire DVD in less than 5 minutes).

I'm also considering buying a PS3 and trying encoding under linux on that for fun...
 
This does sound cool. Ive just done some googling and my initial findings suggest that transcode can be used for this purpose.
 
Last edited:
Out of interest I'm currently reinstalling fedora 8 on my test box to try something. I was getting a nice respectable 300ish fps with handbrake before, but the installation was a general fedora install with the GUI and loads of services.

I'm currently stripping it down to only what's need and I may custom compile the kernel if I can be bothered, anybody care to guess the performance boost that I'll get...?

EDIT: Transcode looks good actually, goes against my plan of using open mosix (which I still like as the same cluster could easily be adapted to ripping my considerable music collection from wav files to high quality mp3...) but well worth a look all the same...
 
Depends what you mean by performance really. I doubt you would see a major increase in encoding performance but you would prob see a benefit to system load times and general responsiveness.
 
True, even with the gui and a bunch of services, it's not likely to make a huge difference to processor power and there's 8GB of RAM in there so it shouldn't be short in that department either. I'm still curious to see what kind of performance I can get out of such a machine purpose built for encoding.

Particularly as the performance from those chips (which are basically conroe Xeons) should be roughly comparably with a QX9650 if I wanted to build a new encoding box.

I'm still research the clustering, though not right now as I'm stuck in telehouse replacing a 10gbit line card....
 
Well the results are in from my rebuild...

Encoding: task 1 of 1, 59.21 % (372.70 fps, avg 370.69 fps, ETA 00h02m42s)

Thats pretty representative of a few runs, the DVD in question was once upon a time in mexico, ripped to ISO and mounted first...Interesting top shows that the system is utilising all the memory available.

top - 03:02:57 up 1:49, 2 users, load average: 3.21, 1.65, 0.82
Tasks: 114 total, 1 running, 113 sleeping, 0 stopped, 0 zombie
Cpu(s): 77.3%us, 1.1%sy, 0.0%ni, 21.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8200244k total, 8154680k used, 45564k free, 129224k buffers
Swap: 2031608k total, 0k used, 2031608k free, 7454844k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1475 root 20 0 261m 91m 2532 S 312 1.1 10:05.26 HandBrakeCLI

Overall I'm quite impressed, I've gone from an average of around 310fps to 370fps by stripping the carp out of the build and compiling Handbrake myself (can't say I think that made a whole heap of difference though).

I tested using the same process on my macbook pro last night (2.16 Ghz) and it averaged about 135fps (CLI version obviously). This suggests that handbrake doesn't scale brilliantly beyond 2 cores (I saw 'only' three and half times the performance on a quad 3Ghz Xeon as opposed to a Dual 2.16Ghz mobile Core Duo).

The next step will be to try a very lightweight linux build I think, the end result is basically that a qx9650 (should be equivilent to the 2x Dual Core 3Ghz Xeon I have - which are conroe based essentially), set up right, can encode a DVD in 7-8 minutes.

I'm still looking at clustering, given that Handbrake appears not to scale brilliantly to additional cores, a set of E8400 based cluster nodes could outperform a qx9650 and would likely be cheaper...
 
You can't run parallel encodes of a video in separate threads or processes as to encode the next chunk the previous chunks need to be done. You could improve performance encoding multiple items at once, but that's about it.
 
Moredhel said:
You can't run parallel encodes of a video in separate threads or processes as to encode the next chunk the previous chunks need to be done. You could improve performance encoding multiple items at once, but that's about it.
Click to expand...

Yeah this is right. I looked for the article on this I read a while ago, but essentially you want to parallize your process as much as possible to gain the best performance increase. It will scalle almost lineraly with processors.

I.e. either write something that splits the movie at key frames ( a frame that doesn't rely on previous frame), and then encode each section on a separate processor, or encode more than one movie at once.

It's like with mp3s, there's no reall way to encode 1 mp3 over multiple processor, indeed the over head would probably wipe out the gain. The speed comes from the fact an album will have say 10-20 tracks, and a quad core could proc 4 tracks at once, or 4 machines could do the same.
 
Is it due to codecs using compression techniques like storing the difference between the current frame and the previous rather than storing the whole frame?
 
It seems so, the only way to do it would be to break the video into sections and do each seperately then join them but with such a short encode time it isn't worth the processing time to join them again...
 
You must log in or register to reply here.
Back
Top Bottom