video encoding software - multi process

bigredshark · 23 Jan 2008 at 22:52

A quick one...

I want to encode some video under linux and I currently use handbrake or mencoder, which works for me but I'd like a little extra performance.

I'm considering setting up a OpenMosix cluster to improve the performance. This spreads load by migrating processes between machines, however handbrake only appears to spawn multiple threads rather than processes.

So, can anybody recommend encoding software which will spawn multiple processes?

bigredshark · 27 Jan 2008 at 17:28

Yes, I suspected mencoder would be the what I needed in the end to do exactly that. It brings me to a bit of a decision though...

My original idea was to spread the encoding of a single file between multiple machines (tricky but should be doable providing I can get something to spawn multiple processes and store the input and output file on a fast NAS.) Then I'd simply encode files one after another (which suits me better as I may want to do one file in a hurry)

I can encode a full DVD (ripped to an ISO file and mounted first) in the quality settings I need in about 15 minutes. (I tested on a 4x 3Ghz Xeon box at work which I wasn't doing anything else with). Given this kind of performance from one big box, the need to distribute the encoding of a single file reduces and I could just encode different files in parallel on each machine.

I'm going to keep looking at it (mainly I have to say because it'd be pretty cool to encode an entire DVD in less than 5 minutes).

I'm also considering buying a PS3 and trying encoding under linux on that for fun...

bigredshark · 28 Jan 2008 at 10:25

Out of interest I'm currently reinstalling fedora 8 on my test box to try something. I was getting a nice respectable 300ish fps with handbrake before, but the installation was a general fedora install with the GUI and loads of services.

I'm currently stripping it down to only what's need and I may custom compile the kernel if I can be bothered, anybody care to guess the performance boost that I'll get...?

EDIT: Transcode looks good actually, goes against my plan of using open mosix (which I still like as the same cluster could easily be adapted to ripping my considerable music collection from wav files to high quality mp3...) but well worth a look all the same...

bigredshark · 28 Jan 2008 at 16:48

True, even with the gui and a bunch of services, it's not likely to make a huge difference to processor power and there's 8GB of RAM in there so it shouldn't be short in that department either. I'm still curious to see what kind of performance I can get out of such a machine purpose built for encoding.

Particularly as the performance from those chips (which are basically conroe Xeons) should be roughly comparably with a QX9650 if I wanted to build a new encoding box.

I'm still research the clustering, though not right now as I'm stuck in telehouse replacing a 10gbit line card....

bigredshark · 29 Jan 2008 at 13:18

Well the results are in from my rebuild...

Encoding: task 1 of 1, 59.21 % (372.70 fps, avg 370.69 fps, ETA 00h02m42s)

Thats pretty representative of a few runs, the DVD in question was once upon a time in mexico, ripped to ISO and mounted first...Interesting top shows that the system is utilising all the memory available.

top - 03:02:57 up 1:49, 2 users, load average: 3.21, 1.65, 0.82
Tasks: 114 total, 1 running, 113 sleeping, 0 stopped, 0 zombie
Cpu(s): 77.3%us, 1.1%sy, 0.0%ni, 21.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8200244k total, 8154680k used, 45564k free, 129224k buffers
Swap: 2031608k total, 0k used, 2031608k free, 7454844k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1475 root 20 0 261m 91m 2532 S 312 1.1 10:05.26 HandBrakeCLI

Overall I'm quite impressed, I've gone from an average of around 310fps to 370fps by stripping the carp out of the build and compiling Handbrake myself (can't say I think that made a whole heap of difference though).

I tested using the same process on my macbook pro last night (2.16 Ghz) and it averaged about 135fps (CLI version obviously). This suggests that handbrake doesn't scale brilliantly beyond 2 cores (I saw 'only' three and half times the performance on a quad 3Ghz Xeon as opposed to a Dual 2.16Ghz mobile Core Duo).

The next step will be to try a very lightweight linux build I think, the end result is basically that a qx9650 (should be equivilent to the 2x Dual Core 3Ghz Xeon I have - which are conroe based essentially), set up right, can encode a DVD in 7-8 minutes.

I'm still looking at clustering, given that Handbrake appears not to scale brilliantly to additional cores, a set of E8400 based cluster nodes could outperform a qx9650 and would likely be cheaper...

bigredshark · 30 Jan 2008 at 09:38

saffyre said:
Why not try gentoo, its perfectly suited to this kind of customisation.

Basically because I'm scared of anything which isn't essentially redhat...

bigredshark · 31 Jan 2008 at 17:47

It seems so, the only way to do it would be to break the video into sections and do each seperately then join them but with such a short encode time it isn't worth the processing time to join them again...