Intel Main Node ↔ AMD Power Server

YiuKorochko

YiuKorochko

YiuKorochko

Associate
Joined
19 Jul 2011
Posts
126
Location
Scotland
Hey all, I've got inspiration for an extremely powerful video encoder and I need you guys' help.

A friend in Canada had something very similar to what I want to set up:

Mid-ranged desktop PC running Windows 7 x64
"Cluster" PC with 42 dual-core CPUs running Solaris x64

Okay, I know how to build a server and install OSs and all that just, but what I need help with is sending a command and video file to the Unix based machine, have it process it, and send the data back the the Windows machine.
The program is x264, written in C so I should be able to send the ops from one OS to another.
Difference is I'll probably be using a quad or dual socket AMD 8 or 12 core.

Can anyone tell me how I'm going to send the commands from the Windows machine to the Unix power server?
 
Last edited:
Vinni3 @H|H said:
I saw your other thread regrading the Intel 980 vs 990x - why not go all out on a SR-2 & Xeon build? Yes, it's old tech by now but still very powerful.

http://www.overclockers.co.uk/showproduct.php?prodid=MB-024-EA&groupid=701&catid=5&subcat=1695

Amazingly, there is still stock of the SR-2 floating around.
Click to expand...

Woah woah, the cheapest I can find these Xeons after a 5 minute search is £150 for a dual core. I can get 12 AMD cores for £200, effectively 48 cores 48 threads versus 8 cores 16 threads. Don't get me wrong, I do prefer Intel over AMD, but AMD offers way more bang/buck. Not too good for high end servers but the raw cores will beat it.
 
Last edited:
So whats the windows machine doing?

Only thing I can think of is the windows machine edits the video and saves it to a shared drive either on the unix box or on a nas. Then you could kick of the encode any number of ways on the unix box.

e.g. SSH to the box and kick of the encoder manually, or setup a script to monitor a directory and fire off the encoder when a file appears.
 
Thing is, I don't know how my old buddy did it but he was able to prove it can be done. The Windows machine sets up the command line for the x264 video encoder and sends it to the server. Once received it kicks into gear, sending the encoder outputs back to the Windows machine.

The way he described the system made it sound pretty much fully automatic, just sends the script and off it goes: he comes back and it's done sitting on his Windows desktop.
It very well could just be a shared HDD with a monitor running on the server waiting for the script but I'm not certain how to set up a shared drive between the computers. I can set up a drive on one PC which allows others to access it, but if I'm going to have to use an HDD as a medium I'd want to wire both of them straight to it.
 
You can't wire them both to it.

You still didn't answer what the point of the windows machine is?
 
Biffa said:
You still didn't answer what the point of the windows machine is?
Click to expand...
Everything except the encoding process.

You do everything else with the Windows machine. The only reason the cheap, no dedicated GPU, no sound card, no accessories server exists is so that you're not spending 30 hours encoding a 1920x1080 BluRay film. You might laugh and say Why and how could that take 30 hours? but if you see the quality of such encodes versus the size they are, it's more than worth it.

More so when it comes to animation. The best animation compression I've done is a 58 minute short film. 1920x1080 23.976FPS with an OGG Stereo audio stream came to 1.07GB. And it's the same quality as the m2ts I ripped from my own copy of the BluRay.

Compressing this video took 8 hours per pass on my Intel i5 750 OCd to 3.5GHz. Now if we look at ratios, then:

4 threads x 3.5GHz = 14 billion OP/s = 16 hours approx.
32 threads x 2GHz = 64 billion OP/s = 46 minutes approx.
 
Last edited:
This is not particularly difficult surely?

Render farms have existed since forever, it's not exactly something off the wall and inspirational you are trying to do here.
 
Last edited:
YiuKorochko said:
Hot-swap HDDs it is then! Good thing my case has one of those.
Click to expand...

Hot-swap HDDs? :confused:

As others have said, it's simple enough to do.

Have a scheduled task on Windows that watches a directory (to which you'll output your files), when a new file arrives, it copies it across to the Unix machine. Unix machine has got a similar watchdog going on, so when a new file appears it then kicks off the command and once finished copies it back over to the Windows machine.
 
MikeHiow said:
Is there not some kind of distributed encoding package? I'd imagine this is what your friend has used.

I did something recently for a local design house who now do their 3D modelling on one machine, but rendering is handled by a powerhouse of a server.

http://www.google.co.uk/search?sourceid=chrome&ie=UTF-8&q=distributed+video+encoding
Click to expand...
Very good search query, this looks very close to what I'm looking for!
Sin_Chase said:
This is not particularly difficult surely?

Render farms have existed since forever, it's not exactly something off the wall and inspirational you are trying to do here.
Click to expand...
No I no it's not difficult, I just don't know how to have the two systems communicate. From what I'm gathering from the above Google search it's pretty much up to the host computer to send the data to the server. The server basically only has the OS and Unix variant of the video encoder and the commands are sent straight towards the Unix executable but the output data is rendered to a network shared drive on the Windows host. Assuming you're using LAN, that seems like a good option but you could also have it render to say, and external HDD hooked up to the server to be safer if WiFi happened to be your only option...

slylittlefox said:
Hot-swap HDDs? :confused:

As others have said, it's simple enough to do.

Have a scheduled task on Windows that watches a directory (to which you'll output your files), when a new file arrives, it copies it across to the Unix machine. Unix machine has got a similar watchdog going on, so when a new file appears it then kicks off the command and once finished copies it back over to the Windows machine.
Click to expand...
Now if I were to program the watchdog myself I think it would be simpler than trying to work out OS built-in commands, but unfortunately I don't know Any Unix based programming so I'll be treading a whole lot of new water with this.

A little less than half the blockade is due to lack of confidence. Not wanting to waste money on something I can't get to work etcetera.

I've got a few ideas brewing in my head now, so I think I'll ask doom9 if MeGUI is capable of utilising a cluster for rendering. Since I haven't talked to my buddy in a few years (I had his e-mail written down but never bothered saving the contact) I don't have any clue which software he used so I'll have to do more detailed research elsewhere.

Thanks for your help guys, sometimes I get stuck when I can't find the right words to Google I suppose. :p


P.S.:
Yes, Hot-swapping HDDs. My Coolermaster 690 II has a SATA-II bay on the top of it so I can plug the file in there and wire it internally to the server before booting it. Then again, if I can figure out how to do it over a network then I won't need to do that. My knowledge of LAN/Network protocols are extremely low. The closest thing I have to that area of things is a little bit of HTML and CSS.
 
Last edited:
NAS box with NFS support
or windows pc exposes its drives via nfs to the unix box
or put samba on the unix host and use smb to windows drives or vice versa
or store the data on the unix box and access it with nfs from the windows pc

any of these will get you going.
 
I'll be very surprised if there is not a package that will handle all this for you, streaming the raw data to the server, encoding it and transferring the rendered product back when it's finished.
 
You must log in or register to reply here.
Back
Top Bottom