In need of some advice and a sanity check before I dip my toes into 10gb home networking..

keenan

keenan

keenan

Associate
Joined
22 Sep 2009
Posts
1,438
My home broadband is getting upgraded to 5gb next Friday, so I'm looking at taking the first steps into upgrading my current 1gb home setup.

My main PC is mini-itx and has onboard 2.5gb Ethernet, but all other home devices are limited to 1gb and WiFi.

I do have a server which handles all media (Plex), ftp and downloads, so ultimately this needs the most attention.

I've done some research by searching this subreddit and so far I've come up with the following plan...

Get an affordable/cheap unmanaged 2.5G switch with 4 x 2.5G Base-T Ports and 2 x 10G SFP+ ports..


Or maybe get a managed switch which supports aggregation in case I ever upgrade my main PC to something with dual 2.5gb Ethernet?


To upgrade the server, I was thinking of getting an Intel x520-da2


Then lastly, to connect the server to the switch's sfp+ port, this cable. (15 meters)



So to recap, new router and server will connect to the 2 x SFP+ ports and main PC with connect to a single 2.5gb port for now.

Are all the items listed above compatible and are they the correct items for this purpose?

Thanks for reading..
 
keenan said:
I've done some research by searching this subreddit
Click to expand...
This ain't Reddit. ;)

What router are you planning to use? I'm assuming copper? What's your planned topology/layout? It sounds awfully like you basically want to hang the server off the 5Gb WAN and everything else can stay as it is. I'd avoid the X520, it's very old now, uses more power than more modern options, and lacks a lot of features and offloads you would fine useful. Look at ConnectX-4Lx or similar, for not much more money and an infinitely better chipset and features. Or if you want Intel and can stretch, an X550.

Why not put one in the server, and one in the mini-ITX PC, and lose the 2.5G cap? You're paying for 5G WAN, so you may as well use it. Personally, I'd bite the bullet now and install a proper 10G spine to make upgrades and local device shuffling painless down the line. I would do something like:

ONT --> x86 router with a ConnectX-4Lx
| DAC uplink​
--> Mikrotik CRS305-1G-4S+ 10G SFP+ switch​
|----> Server and PC (AOC or similar)​
| DAC uplink​
----> Cheap Chinese 2.5G 8 port copper switch with 1x SFP+ uplink (with PoE if needed for WiFI AP)​
----> Legacy 1G/2.5G copper clients​
​
That way you can have 5Gb WAN coming in to a 10G router, a full 10G core switch and one or more cheap copper 2.5G switches with an SFP+ uplink back to the core switch for legacy clients. If you want to go budget and just get faster speeds on your server, leave whatever router the ISP gives you and the switch you linked will work. Be careful of planning ports though as you'll need to use one SFP+ port to uplink to the router with an RJ45-SFP+ module (else you'll lose the 5G speeds) and one for your server... and then you're out of options.
 
Last edited:
@Rainmaker

Thank you for the detailed reply! Definitely something Reddit doesn't provide! :)

Regarding the main mini-ITX pc, there's no option for expansion, so I'm very much stuck with 2.5G. Unless I try some form of usb to ethernet solution?

In regards to the X520.. apart from using more power, what sort of features would I loose out on compared to something newer?
 
Seems like a waste of a 5 Gb connection. I have two TP-Link TL-ST1008 10 Gb 8 port switches - one in my study, one in my 'server room' - both modded with quiet Noctua fans and Intel 10 Gb NICs in all devices. I'm sure there are cheaper devices now. All RJ45.

keenan said:
Regarding the main mini-ITX pc, there's no option for expansion, so I'm very much stuck with 2.5G. Unless I try some form of usb to ethernet solution?
Click to expand...

You can get a M.2 to 10 Gb ethernet adapter.
 
Quartz said:
Seems like a waste of a 5 Gb connection. I have two TP-Link TL-ST1008 10 Gb 8 port switches - one in my study, one in my 'server room' - both modded with quiet Noctua fans and Intel 10 Gb NICs in all devices. I'm sure there are cheaper devices now. All RJ45.



You can get a M.2 to 10 Gb ethernet adapter.
Click to expand...
Apologies, I should have mentioned that both M.2 slots are being used for storage.
 
Speaking in generalisations and from experience doing this myself.
Stay away from copper as much as you can, it is often more expensive for the modules and uses a lot more power then optics.
Intel SFP cards can be picky with SFP's. You may find generic cheap DAC or AOC won't work, if you can try to find preloved units with SFP modules.
Switching is expensive.

Initially I did the 10Gb link from Core to Edge, then from Core to Firewall and finally clients. I have maybe 3 clients connected via fibre directly and then another client on a 4 1Gb interface LACP connection. It's probably been a huge waste of time and money and it haunts me now as newer and shinier kit is coming out that I now can't get into as the entry pricing is so high as I refuse to back down to 1Gb.
 
Quartz said:
Is freeing up a slot by adding a SATA SSD an option?
Click to expand...
After some thinking I believe this is the best option.

I'll free up the second m.2 slot and connect that drive via an external 10Gbps enclosure. That should be more than adequate for gaming.
 
Most of this has already been answered, but for the sake of clarity, XA520 doesn’t do NBASE-T, so no 2.5Gb/5Gb, it’s 10Gb or 1Gb with nothing in between, you need the 550 for NBASE-T or a 7 series.

The Intel white listing of optics is trivial to change, either at a driver level or preferably a firmware level, takes 2 minutes, then works with anything. The CX4 however is inexpensive and just works, it’s also efficient compared to the earlier Intel stuff.

5Gb USB NIC’s are a thing, that may be a better option/cheaper than the M.2 option/external enclosure.

Personally, SFP+ and fiber or DAC is the better option, I wouldn’t want 5Gb WAN and not to be able to use it fully on my PC/servers, and that likely means spending some money. I went UDM-SE, used the Mikrotik copper to SFP+ adapter (it runs hot!), then SFP+ fiber to my core Aruba 1930, which then feeds fiber to each floor and uses a switch similar to the one you linked to in order to break that out into copper at 2.5Gb for legacy clients. Future client upgrades are fine for 2.5Gb copper or SFP+ 10Gb, if I need more on the copper side (eg 5Gb) I swap out a switch. I can also upgrade some clients using USB 3 2.5Gb NIC’s, I used to call out RealTek for making software NIC’s where they would nerf the hardware/firmware and literally do it all in software on some chipsets, but they seem to have improved a lot under both Linux and Windows, and at least in the 2.5Gb class, they were a safer bet than Intel for several years.
 
Thank you for the info @Avalon

I decided (hopefully correctly) to go for two Connect-X4 CX4121C cards for the server and main PC.

For the main PC I'll use an M.2 to PCIe 4.0 X 4 riser. No additional steps needed on the server side.

The pc and server will have a direct connection between each other via one port/link and then to the switch via copper and sfp+ adapter from the other port/link.

Hopefully my calculations and assumptions are all correct and I'm not headed for complete disaster.

Ideally I would've liked an affordable 10gb switch with 3 or 4 sfp+ ports and even just a single rj45 to link to my old gigabit switch for all older devices.
 
Dumb question, you have 5Gb WAN, but it’s getting fed to the switch via 2.5Gb, you understand that the only 10Gb connection is between the server and the main PC and they’ll never see more than 2.5Gb via WAN unless - admittedly just re-woke up - I am missing something?
 
Avalon said:
Dumb question, you have 5Gb WAN, but it’s getting fed to the switch via 2.5Gb, you understand that the only 10Gb connection is between the server and the main PC and they’ll never see more than 2.5Gb via WAN unless - admittedly just re-woke up - I am missing something?
Click to expand...
Yes, 5gb wan fed to the switch via single 10gb rj45 on the router (all other ports are 1gb). This will be copper to sfp+ adapter going into the switch. Then sfp+ DAC from the switch to server. For the main PC I'll use the onboard 2.5gb for internet and a direct 25gb sfp+ link to the server for all other tasks.

Unless I can negate the 2.5gb and get internet shared from the server?? Is that still a thing?
 
Last edited:
5Gb feed into your router. What are you using router to switch? You can't just jack your WAN into your switch and watch all devices leverage that speed.
 
Bite the bullet and buy a 10Gb capable switch, if you go copper you can probably get 10Gb off of all your existing wiring and you can ultimately feed all the things with 5Gb. I have a few ~30m cable runs doing 10Gb easy.
 
Last edited:
sandys said:
Bite the bullet and buy a 10Gb capable switch, if you go copper you can probably get 10Gb off of all your existing wiring and you can ultimately feed all the things with 5Gb. I have a few ~30m cable runs doing 10Gb easy.
Click to expand...
@sandys

You mean 10Gb rj45 switch? I checked online and they seemed quite expensive compared to sfp+.

I will only ever need a fast connection for the 2 PC's mentioned, there rest is all old antiques. :)
Steveocee said:
5Gb feed into your router. What are you using router to switch? You can't just jack your WAN into your switch and watch all devices leverage that speed.
Click to expand...
The supplied router only has 1 x 10gb rj45 lan port, 1 x 10gb sfp+ wan port and the rest are all 1gb ports.

I'll be using a cat7 cable with a 10GBase-T SFP+ rj45 transceiver to connect the rj45 end to the router and rj45->sfp+ to the switch.

As far as I'm aware that's the only affordable way to get the 5Gb from the router converted to sfp+ before going into the switch.

Unless there's a 10gb switch somewhere with both rj45 and sfp+ ports?
 
You must log in or register to reply here.
Back
Top Bottom