Home Server + 10G ethernet

[Kei]

Small update. It ran faultlessly since completion, up until the end of last week where something occurred with an update that broke samba. It seems that somehow a repository for a newer version of opensuse (42) added itself which then caused an incompatible version to install on my system running 13.1. I probably could have fixed it but it was far easier to just download the latest 42.2 iso and reinstall the os. Wiping the SSD and reinstalling took about half an hour. A few tweaks to the config, added repositories and programs and it was back to the way it was before the muck up. The array remounted to the same mount point and shared via samba and plex media server working as it should.

I went with straight XFCE instead of gnome or KDE this time. Something has made a fair difference as it uses 1GB less ram than before. Might be down to the fact that I've not yet reinstalled lsi megaraid storage manager.

 

[Kei]

Happy to report that all is still running well. I replaced the 2x2GB of corsair memory with the old kingston 4x4GB sticks from my old x79 system. I also updated to OpenSUSE 42.3 today.


I've now updated the parents pc to my old x79 system, so their 990FX motherboard and FX8320 are now sitting unused. I might well be swapping these parts into the server and finally retiring the phenom II. The benefit will obviously be 8 cores vs 4, but there is also built in usb 3.0 and the additional full size pcie slots meaning that the sas expander card can fit into the board and I can use the side panel fan to help cool the sas cards.

I also noticed that WD no longer make the Se enterprise disks so I'm assuming that I'll have to look at gold if I want/need any more disks.

 

[WantoN]

Loved seeing how this progresses over the years, nice on mate.

I plan on putting my rig onto server duties (mainly backup and streaming) when I upgrade. If cannonlake is good enough, I’ll do it then.

Any advice for newbies?

 

[Kei]

Undervolting and ensuring all power saving features for the pic are enabled help to minimise power consumption. An integrated GPU is the most ideal solution but a low power discrete GPU like I’ve got isn’t too much worse.

Preventing the disks from spinning down is beneficial to their lifespan even though it is at the cost of power consumption. The big investment in these kind of projects is in the disks. The hardware in mine cost around £250, mostly in the sas expander and the disk back planes. The disks are nearly £800.

In my case, i chose to go with hardware raid using an lsi sas card that I managed to get free. Most other people seem to opt for a cheaper software based raid solution. I don’t have any experience with these methods.

 

[GodAtum]

Are you planning to use Freenas?

 

[Kei]

No point in freenas in my case as I use hardware based raid. Freenas is a software based raid that uses ZFS. To use ZFS, I’d have to force the raid controller card into IT mode (initiator target IIRC) which presents all the individual disks to the os rather than an array as a single giant disk.

 

[GodAtum]

Ah OK. Are you just using Samba or NFS to serve the shares?

 

[Kei]

Samba.

 

[AiiR]

Have you measured the power consumption at all? When I built my home file server this was my primary concern, which is why I chose an AM1 board/cpu as it was super cheap and easy on the wattage. Would be interesting to see how a Phenom fairs.

 

[Kei]

Sadly I still haven't invested in a power meter in order to monitor it. Based on the idle usage in the house overnight, it can't be too bad but it would be nice to know exactly as I under-volted and dropped the multiplier in order to reduce the power consumption. I reckon that the hardware raid card, sas expander card and the use of 7x enterprise 7200 rpm disks will probably mean it's got to be in the 100W range at idle.

 

[Frozennova]

Have you measured the power consumption at all? When I built my home file server this was my primary concern, which is why I chose an AM1 board/cpu as it was super cheap and easy on the wattage. Would be interesting to see how a Phenom fairs.

My Home/Server NAS of reasonably similar spec (FX4100, 16GB DDR3, Dell SAS6/ir, 8 WD Reds, 1 SSD) idles somewhere in the 70w range

 

[skylineboy]

Excellent work as usual Kei!

 

[Kei]

All still running well. Nice and cool with the current winter temperatures. Starting to look at adding more disks (WD ultrastars as the Se is no longer made) and also considering an array rebuild as the stripe size of 256kb means that some folders stored on there are taking up vast amounts of space unnecessarily.

 

[Webzta]

Did you consider running drivepool? Cheap, flexible, can utilise ssd cache etc.

https://stablebit.com/

 

[Kei]

Did you consider running drivepool? Cheap, flexible, can utilise ssd cache etc.

https://stablebit.com/

Nope. I’d have needed a different hba that could be flashed to support it mode so that each disk connected to it could be seen individually by the OS. This project was designed to use up all of my spare parts and freebies meaning my only investment in it was the actual disks for the array, the sas expander card, power supply, backplanes and a few cables.

 

[Kei]

Doing networking upgrades to help consolidate the existing network down.

Cisco VA-DSL-A EHWIC fitted to my 1921 router to remove the ECI openreach modem. First one I received was faulty. (no modem PHY or firmware) The replacement was perfect.



It's nice having stats for the modem.

Controller VDSL 0/0/0 is UP

Daemon Status:        Up

            XTU-R (DS)        XTU-C (US)
Chip Vendor ID:        'BDCM'            'IFTN'
Chip Vendor Specific:   0x0000            0xB206
Chip Vendor Country:    0xB500            0xB500
Modem Vendor ID:    'CSCO'            '    '
Modem Vendor Specific:  0x4602            0x0000
Modem Vendor Country:   0xB500            0x0000
Serial Number Near:    FOC16390RTR 1921/K9 15.7(3)M2
Serial Number Far:     5501570926
Modem Version Near:    15.7(3)M2
Modem Version Far:     0xb206

Modem Status:        TC Sync (Showtime!)

DSL Config Mode:    AUTO
Trained Mode:    G.993.2 (VDSL2) Profile 17a
TC Mode:        PTM
Selftest Result:    0x00
DELT configuration:    disabled
DELT state:        not running

Full inits:        2
Failed full inits:    0
Short inits:        0
Failed short inits:    1

Firmware    Source        File Name
--------    ------        ----------
VDSL        embedded      VDSL_LINUX_DEV_01212008

Modem FW  Version:    4.14L.04A
Modem PHY Version:    A2pv6C039t.d24o_rc1
Trellis:        ON              ON
SRA:             disabled        disabled
 SRA count:         0            0
Bit swap:         enabled        enabled
 Bit swap count:    1441            175
Line Attenuation:    13.1 dB          0.0 dB
Signal Attenuation:      0.0 dB          0.0 dB
Noise Margin:          6.5 dB          6.7 dB
Attainable Rate:    66474 kbits/s        21532 kbits/s
Actual Power:          0.7 dBm          0.6 dBm
Per Band Status:          D1     D2     D3     U0     U1     U2     U3
Line Attenuation(dB):   8.1    18.8    30.9    0.1    14.8    23.0    N/A  
Signal Attenuation(dB): 10.1    18.6    30.9    0.1    14.6    22.9    N/A  
Noise Margin(dB):       6.5    6.5    6.5    6.4    7.1    6.6    N/A  
Total FECC:        0            6545
Total ES:        107            96
Total SES:        15            13
Total LOSS:        1            0
Total UAS:        83            68
Total LPRS:        0            0
Total LOFS:        10            0
Total LOLS:        0            0


          DS Channel1      DS Channel0    US Channel1      US Channel0
Speed (kbps):              0           64159             0           18999
SRA Previous Speed:       0               0             0               0
Previous Speed:              0           63650             0           18999
Reed-Solomon EC:          0               0             0              36
CRC Errors:              0            8805             0             810
Header Errors:              0               3             0               0
Interleave (ms):       0.00            0.00          0.00            0.00
Actual INP:           0.00            0.00          0.00            0.00

Training Log :    Stopped
Training Log Filename :    flash:vdsllog.bin

Looking to consolidate the rest by consolidating the GS108 switch into a unit with POE on board to power the access points and whilst I'm at it add in some 10Gbe ports. I started looking at the MikroTik offerings with the CRS328-24P-4S+RM standing out as the most ideal option. Considering the facilities on offer, it seemed a little too good to be true based on the cost. Reviews for it were scant and what I could find seemed a bit hit and miss. I decided to look for something higher end on the used market fitting the same specs. (8 or more 1Gbe POE+ports, 4 or more SFP+ ports) Narrowed it down to an Aruba S2500-24P, Dell X1052P and a Juniper EX3300. The Aruba is pretty hard to find in the UK and the Dell was likely to cost around £500 for a refurb. The Juniper can be had for £360 for an open box new item, easily covers all my needs and is enterprise proven unlike the mikrotik. It might be a little noisier and a tad greedier on power but it's a known quantity and if I ever have need of its layer 3 features, it'll thoroughly outgun the mikrotik.


It'll be slotted in my mini home made rack with the cisco router and 12 way MDU.


A basic network diagram with the 10Gbe tweak. A few devices are missing off this, but it's got the bulk of it.


The server itself will need the replacement 990FX motherboard that I have spare in order to support a 10Gbe ethernet card as my old 790FX motherboard lacks the necessary number of PCIe lanes. I'll need to decide whether to also use the FX8320 or stick with the old phenom II x4. I need to decide on the SFP+ network adaptors I want to use. The usual choice seems to be mellanox connect-X items. As usual, these seem to be more common in the US than the UK. I've seen HP ones about but not sure how they fair.

I've also been considering moving from opensuse to centos when I change the hardware as I've been unable to get the LSI stoage manager server to install properly on opensuse since moving to leap. I installed centos 7 on an old server in work to use as a floating license server and found it pretty similar in most aspects but with better support for some software and a more active community.

 

[Kei]

It has arrived. One Juniper EX3300 24P switch.

 

[Kei]

Fitted and configured. This thing was an absolute steal considering it was brand new and has a september 2018 build of junos loaded. (15.1R7.9)


Now I need to find some SFP+ NICs. I had been looking at trying to find a mellanox connectx 2 or 3 card but they seem to be common in the US and china. The only cards that I've come across that suits my needs are intel x520 and a few solarflare models. Having done a lot of research, it seems the intel cards are probably my best bet, though I am likely limited to intel coded SFP's which isn't too much of an issue with the fibre transceivers. The DAC may be different if the juniper doesn't like an intel coded DAC and the intel NIC doesn't like a juniper coded DAC.

 

[BongoHunter]

These are available in UK for £36 refurbed:

Mellanox MNPA19-XTR Single Port - 10GbE SFP+ Full Height PCIe-x8 Ethernet

You just need to Google quite hard to find them...

 

[Kei]

Upgrade to the spare 990FX board with an FX8320 in it.

New board ready to go in. Old 790FX board and phenom II to come out. 4x4GB memory sticks are coming out to go into the x79 system I gave to my parents and their 2x8GB sticks are going in this. I'm going to try and squeeze the 2x2GB corsair sticks in if I can but I'm not sure if they will fit with the cooler.


Things don't look too bad inside considering it's been in the loft for 4.5 years and I've never dusted it. I've checked it over every couple of months but it's never looked too bad. The dust filters were looking a bit sorry for themselves.


In other news, I've purchased one mellanox connectx-3 Pro EN and two intel X710-DA2 10Gbe SFP+ NIC's.