Home Server + 10G ethernet

A full network rebuild has been performed as my old tp link 1043nd router has been requiring a prod every couple of days as it would just freeze up and stop all network traffic. I decided that i was up for a challenge and went with a cisco 1921/K9 ISR, an hp 1910-8G managed switch and a D-link DAP-2695 wireless access point. Complete and total overkill, I know, but I got them at a good price and being enterprise kit they ought to be reliable. (which was my main consideration) 5GHz wifi certainly is a lot quicker than 2.4 and the broadcast range from the d-link seems to travel slightly further too.
15557595940_e220ffbabc_o.jpg

15534637309_6777691a7e_o.jpg


Having lived with it for just over a week, I can say that it has behaving well. (current PPPoE session has been up for 4 days so far, which is better than the tp link would usually manage) I certainly get better performance from it too. Very pleased there. (used to get low-mid 74 down and high 17- low 18 up, ping is about the same)
3907668415.png


Config wise, it took me about 4 hours to configure the router sufficiently that i could access the internet. It took me a few days more and some assistance of the helpful network gurus to sort out the necessary static nat and access lists to get the server to be accessible from the outside for ftp, ssh and teamspeak. Still not sure on steam games as I don't do multiplayer very often. I'm not sure if I've covered all bases on the deny list but I'm hopeful that it's close.

The basic config for running with a bt infinity vdsl2 modem for those that are interested.
Code:
!
! Last configuration change at 12:44:44 UTC Tue Nov 11 2014
version 15.2
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname Cisco-1921
!
boot-start-marker
warm-reboot
boot-end-marker
!
!
enable secret 5.
enable password 7
!
no aaa new-model
!
ip cef
!
!
!!!!!!DHCP server config
ip dhcp excluded-address 192.168.0.1
ip dhcp excluded-address 192.168.0.50 192.168.0.51
!
ip dhcp pool main_dhcp_pool
 network 192.168.0.0 255.255.255.0
 default-router 192.168.0.1
 dns-server x.x.x.x
!
ip dhcp pool Kei-PC
 host 192.168.0.2 255.255.255.0
 client-identifier xxx
 client-name Kei-PC
!
ip dhcp pool Humax
 host 192.168.0.4 255.255.255.0
 client-identifier xxx
 client-name Humax
!
ip dhcp pool Kei-NAS
 host 192.168.0.3 255.255.255.0
 hardware-address xxx
 client-name Kei-NAS
!
ip dhcp pool Server
 host 192.168.0.100 255.255.255.0
 hardware-address xxx
!
!
!
ip domain name WesNet
ip name-server 8.8.8.8
no ipv6 cef
multilink bundle-name authenticated
!
!
!
license udi pid CISCO1921/K9 sn xxx
!
!!!!!Archive every 7 days or on writing config
archive
 path ftp:/xxx/Cisco-1921
 write-memory
 time-period 10080
!
!
!
!
!
!
interface Embedded-Service-Engine0/0
 no ip address
 shutdown
!
!
!!!!!! Note that Ge0/0 has no IP address
interface GigabitEthernet0/0
 no ip address
 ip tcp adjust-mss 1452
 duplex auto
 speed auto
 pppoe enable group global
 pppoe-client dial-pool-number 1
!
!
!!!!!! This is the LAN side
interface GigabitEthernet0/1
 ip address 192.168.0.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
 ip tcp adjust-mss 1452
 duplex auto
 speed auto
 no mop enabled
!
!!!!!! The ISP's given IP address will be configured via d1
interface Dialer1
 ip address negotiated
 ip access-group 101 in
 no ip unreachables
 ip mtu 1492
 ip nat outside
 ip virtual-reassembly in
 encapsulation ppp
 dialer pool 1
 ppp chap hostname [email protected]
 ppp chap password 7 xxx
 ppp pap sent-username [email protected] password 7 xxx
!
ip forward-protocol nd
!
ip http server
no ip http secure-server
!
!!!!!! This is the dynamic PAT between Dialer1 (WAN) interface IP address
!!!!!! and local IP addresses within ACL 1
ip nat inside source list 1 interface Dialer1 overload
ip nat inside source static tcp 192.168.0.3 20 interface Dialer1 20
ip nat inside source static tcp 192.168.0.3 21 interface Dialer1 21
ip nat inside source static tcp 192.168.0.3 22 interface Dialer1 22
ip route 0.0.0.0 0.0.0.0 Dialer1
ip route 192.168.0.0 255.255.255.0 GigabitEthernet0/1
!
access-list 1 remark INTERNET-ACCESS
access-list 1 permit 192.168.0.0 0.0.0.255
!
access-list 101 remark DENY FAKE IPs
access-list 101 deny   ip 192.168.0.0 0.0.255.255 any
access-list 101 deny   ip 172.16.0.0 0.15.255.255 any
access-list 101 deny   ip 10.0.0.0 0.255.255.255 any
access-list 101 deny   ip 127.0.0.0 0.255.255.255 any
access-list 101 deny   ip 255.0.0.0 0.255.255.255 any
access-list 101 deny   ip 248.0.0.0 7.255.255.255 any
access-list 101 deny   ip 224.0.0.0 7.255.255.255 any
!
access-list 101 remark DENY SPOOFING IPs
access-list 101 deny   ip host 0.0.0.0 any
access-list 101 deny   ip host 255.255.255.255 any
!
access-list 101 remark DENY VULNERABLE PORTS
access-list 101 deny tcp any any range 135 139 log-input
access-list 101 deny udp any any range 135 139 log-input
access-list 101 deny tcp any any eq 2000 log-input
access-list 101 deny tcp any any eq 2001 log-input
access-list 101 deny tcp any any eq 6000 log-input
access-list 101 deny tcp any any eq 6001 log-input
access-list 101 deny tcp any any range 5900 5910 log-input
access-list 101 deny tcp any any range 5800 5810 log-input
access-list 101 deny tcp any any eq finger log-input
!
access-list 101 remark DENY TRACEROUTE OUTSIDE
access-list 101 deny udp any any range 33400 34400 log-input
!
access-list 101 remark ALLOW ACCESS FOR SERVER FTP & SSH
access-list 101 permit tcp any any eq 20
access-list 101 permit tcp any any eq 21
access-list 101 permit tcp any any eq 22
!
access-list 101 permit udp any any eq ntp
!
access-list 101 permit icmp any any net-unreachable
access-list 101 permit icmp any any host-unreachable
access-list 101 permit icmp any any port-unreachable
access-list 101 permit icmp any any packet-too-big
access-list 101 permit icmp any any administratively-prohibited
access-list 101 permit icmp any any source-quench
access-list 101 permit icmp any any ttl-exceeded
access-list 101 permit icmp any any echo-reply
access-list 101 deny icmp any any
access-list 101 permit tcp any any established
access-list 101 permit udp any any
!
!
!
!
snmp-server community WesNet RO
snmp-server enable traps entity-sensor threshold
!
control-plane
!
!
!
line con 0
line aux 0
line 2
 no activation-character
 no exec
 transport preferred none
 transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
 stopbits 1
line vty 0 4
 password 7
 login
 transport input all
!
scheduler allocate 20000 1000
!
end
 
Last edited:
Well with most of summer gone it seems the server barely even broke a sweat in the loft. Not seen higher than 50 degrees out of it so far, but then I'm hardly taxing the cpu. RAID array is still in a nice healthy state with all 7 WD Se disks behaving as they should. Considering upgrading my LSI 8888elp to something with newer standards that would allow me to attach an LTO tape drive to it and run LTFS. Presently I can run a tape drive but I can't run LTFS as the HBA doesn't support it. Backup software like symantec backup exec would work but it's not as convenient and I'm not even sure you can get it or an equivalent for opensuse.

No idea on the power consumption as yet, need to consult the bills and run a comparison as we're nearly a year down the road.
 
Picked up a replacement GPU for the server. A nice tidy nvidia quadro NVS 310 with a measly 19.5W maximum power draw. The old FX3500 was a greedy card at 95W which was complete overkill for server purposes.
NVS310 by Kyle, on Flickr

I finally got VNC to work properly. It turned out that gnome 3 and even gnome classic DE's wouldn't play nicely with VNC due to hardware rendering being required. I've now switched to using XFCE which works perfectly. I've also switched the login screen to lightdm which is easier on the hardware too. Space is getting chomped up quite quickly due to the expansion of my plex library. I think my handbrake encode quality might be a tad high as some of the bluray encodes are well above 20GB a piece.
Home Server by Kyle, on Flickr
 
Essentially, it usually saturates the gigabit network link at 100MB/s, though I have seen it fall as low as 32 on occasion. I think my combination of stripe size and cluster size on the ext4 file system makes write speeds on smaller files slightly lower.
 
Small update. It ran faultlessly since completion, up until the end of last week where something occurred with an update that broke samba. It seems that somehow a repository for a newer version of opensuse (42) added itself which then caused an incompatible version to install on my system running 13.1. I probably could have fixed it but it was far easier to just download the latest 42.2 iso and reinstall the os. Wiping the SSD and reinstalling took about half an hour. A few tweaks to the config, added repositories and programs and it was back to the way it was before the muck up. The array remounted to the same mount point and shared via samba and plex media server working as it should.

I went with straight XFCE instead of gnome or KDE this time. Something has made a fair difference as it uses 1GB less ram than before. Might be down to the fact that I've not yet reinstalled lsi megaraid storage manager.
 
Last edited:
Happy to report that all is still running well. I replaced the 2x2GB of corsair memory with the old kingston 4x4GB sticks from my old x79 system. I also updated to OpenSUSE 42.3 today.


I've now updated the parents pc to my old x79 system, so their 990FX motherboard and FX8320 are now sitting unused. I might well be swapping these parts into the server and finally retiring the phenom II. The benefit will obviously be 8 cores vs 4, but there is also built in usb 3.0 and the additional full size pcie slots meaning that the sas expander card can fit into the board and I can use the side panel fan to help cool the sas cards.

I also noticed that WD no longer make the Se enterprise disks so I'm assuming that I'll have to look at gold if I want/need any more disks.
 
Last edited:
Undervolting and ensuring all power saving features for the pic are enabled help to minimise power consumption. An integrated GPU is the most ideal solution but a low power discrete GPU like I’ve got isn’t too much worse.

Preventing the disks from spinning down is beneficial to their lifespan even though it is at the cost of power consumption. The big investment in these kind of projects is in the disks. The hardware in mine cost around £250, mostly in the sas expander and the disk back planes. The disks are nearly £800.

In my case, i chose to go with hardware raid using an lsi sas card that I managed to get free. Most other people seem to opt for a cheaper software based raid solution. I don’t have any experience with these methods.
 
No point in freenas in my case as I use hardware based raid. Freenas is a software based raid that uses ZFS. To use ZFS, I’d have to force the raid controller card into IT mode (initiator target IIRC) which presents all the individual disks to the os rather than an array as a single giant disk.
 
Last edited:
Sadly I still haven't invested in a power meter in order to monitor it. Based on the idle usage in the house overnight, it can't be too bad but it would be nice to know exactly as I under-volted and dropped the multiplier in order to reduce the power consumption. I reckon that the hardware raid card, sas expander card and the use of 7x enterprise 7200 rpm disks will probably mean it's got to be in the 100W range at idle.
 
All still running well. Nice and cool with the current winter temperatures. Starting to look at adding more disks (WD ultrastars as the Se is no longer made) and also considering an array rebuild as the stripe size of 256kb means that some folders stored on there are taking up vast amounts of space unnecessarily.
 
Did you consider running drivepool? Cheap, flexible, can utilise ssd cache etc.

https://stablebit.com/
Nope. I’d have needed a different hba that could be flashed to support it mode so that each disk connected to it could be seen individually by the OS. This project was designed to use up all of my spare parts and freebies meaning my only investment in it was the actual disks for the array, the sas expander card, power supply, backplanes and a few cables.
 
Doing networking upgrades to help consolidate the existing network down.

Cisco VA-DSL-A EHWIC fitted to my 1921 router to remove the ECI openreach modem. First one I received was faulty. (no modem PHY or firmware) The replacement was perfect.



It's nice having stats for the modem.
Code:
Controller VDSL 0/0/0 is UP

Daemon Status:        Up

            XTU-R (DS)        XTU-C (US)
Chip Vendor ID:        'BDCM'            'IFTN'
Chip Vendor Specific:   0x0000            0xB206
Chip Vendor Country:    0xB500            0xB500
Modem Vendor ID:    'CSCO'            '    '
Modem Vendor Specific:  0x4602            0x0000
Modem Vendor Country:   0xB500            0x0000
Serial Number Near:    FOC16390RTR 1921/K9 15.7(3)M2
Serial Number Far:     5501570926
Modem Version Near:    15.7(3)M2
Modem Version Far:     0xb206

Modem Status:        TC Sync (Showtime!)

DSL Config Mode:    AUTO
Trained Mode:    G.993.2 (VDSL2) Profile 17a
TC Mode:        PTM
Selftest Result:    0x00
DELT configuration:    disabled
DELT state:        not running

Full inits:        2
Failed full inits:    0
Short inits:        0
Failed short inits:    1

Firmware    Source        File Name
--------    ------        ----------
VDSL        embedded      VDSL_LINUX_DEV_01212008

Modem FW  Version:    4.14L.04A
Modem PHY Version:    A2pv6C039t.d24o_rc1
Trellis:        ON              ON
SRA:             disabled        disabled
 SRA count:         0            0
Bit swap:         enabled        enabled
 Bit swap count:    1441            175
Line Attenuation:    13.1 dB          0.0 dB
Signal Attenuation:      0.0 dB          0.0 dB
Noise Margin:          6.5 dB          6.7 dB
Attainable Rate:    66474 kbits/s        21532 kbits/s
Actual Power:          0.7 dBm          0.6 dBm
Per Band Status:          D1     D2     D3     U0     U1     U2     U3
Line Attenuation(dB):   8.1    18.8    30.9    0.1    14.8    23.0    N/A  
Signal Attenuation(dB): 10.1    18.6    30.9    0.1    14.6    22.9    N/A  
Noise Margin(dB):       6.5    6.5    6.5    6.4    7.1    6.6    N/A  
Total FECC:        0            6545
Total ES:        107            96
Total SES:        15            13
Total LOSS:        1            0
Total UAS:        83            68
Total LPRS:        0            0
Total LOFS:        10            0
Total LOLS:        0            0


          DS Channel1      DS Channel0    US Channel1      US Channel0
Speed (kbps):              0           64159             0           18999
SRA Previous Speed:       0               0             0               0
Previous Speed:              0           63650             0           18999
Reed-Solomon EC:          0               0             0              36
CRC Errors:              0            8805             0             810
Header Errors:              0               3             0               0
Interleave (ms):       0.00            0.00          0.00            0.00
Actual INP:           0.00            0.00          0.00            0.00

Training Log :    Stopped
Training Log Filename :    flash:vdsllog.bin

Looking to consolidate the rest by consolidating the GS108 switch into a unit with POE on board to power the access points and whilst I'm at it add in some 10Gbe ports. I started looking at the MikroTik offerings with the CRS328-24P-4S+RM standing out as the most ideal option. Considering the facilities on offer, it seemed a little too good to be true based on the cost. Reviews for it were scant and what I could find seemed a bit hit and miss. I decided to look for something higher end on the used market fitting the same specs. (8 or more 1Gbe POE+ports, 4 or more SFP+ ports) Narrowed it down to an Aruba S2500-24P, Dell X1052P and a Juniper EX3300. The Aruba is pretty hard to find in the UK and the Dell was likely to cost around £500 for a refurb. The Juniper can be had for £360 for an open box new item, easily covers all my needs and is enterprise proven unlike the mikrotik. It might be a little noisier and a tad greedier on power but it's a known quantity and if I ever have need of its layer 3 features, it'll thoroughly outgun the mikrotik.


It'll be slotted in my mini home made rack with the cisco router and 12 way MDU.


A basic network diagram with the 10Gbe tweak. A few devices are missing off this, but it's got the bulk of it.


The server itself will need the replacement 990FX motherboard that I have spare in order to support a 10Gbe ethernet card as my old 790FX motherboard lacks the necessary number of PCIe lanes. I'll need to decide whether to also use the FX8320 or stick with the old phenom II x4. I need to decide on the SFP+ network adaptors I want to use. The usual choice seems to be mellanox connect-X items. As usual, these seem to be more common in the US than the UK. I've seen HP ones about but not sure how they fair.

I've also been considering moving from opensuse to centos when I change the hardware as I've been unable to get the LSI stoage manager server to install properly on opensuse since moving to leap. I installed centos 7 on an old server in work to use as a floating license server and found it pretty similar in most aspects but with better support for some software and a more active community.
 
Last edited:
Fitted and configured. This thing was an absolute steal considering it was brand new and has a september 2018 build of junos loaded. (15.1R7.9)


Now I need to find some SFP+ NICs. I had been looking at trying to find a mellanox connectx 2 or 3 card but they seem to be common in the US and china. The only cards that I've come across that suits my needs are intel x520 and a few solarflare models. Having done a lot of research, it seems the intel cards are probably my best bet, though I am likely limited to intel coded SFP's which isn't too much of an issue with the fibre transceivers. The DAC may be different if the juniper doesn't like an intel coded DAC and the intel NIC doesn't like a juniper coded DAC.
 
Upgrade to the spare 990FX board with an FX8320 in it.

New board ready to go in. Old 790FX board and phenom II to come out. 4x4GB memory sticks are coming out to go into the x79 system I gave to my parents and their 2x8GB sticks are going in this. I'm going to try and squeeze the 2x2GB corsair sticks in if I can but I'm not sure if they will fit with the cooler.


Things don't look too bad inside considering it's been in the loft for 4.5 years and I've never dusted it. I've checked it over every couple of months but it's never looked too bad. The dust filters were looking a bit sorry for themselves.


In other news, I've purchased one mellanox connectx-3 Pro EN and two intel X710-DA2 10Gbe SFP+ NIC's.
 
I managed to squeeze the 2x2GB corsair sticks in but had to remove the fins from the one module to get it to fit under the cooling fan. The intel expander card is now in the board which makes things much tidier. I can also now fit the side panel fan in place to aid cooling the cards.


Back side looks reasonably tidy.


It initially wouldn't boot up as it didn't like the memory config. Tried upping the DRAM voltage and tried dropping to 1333 (from 1600) but no dice, I had to run 1066 in order to get it to boot. Might be possible to run it all at 1600 by tweaking the timings but I'm not sure it's worth it in this usage case. Linux seemed to cope ok with the changes, however I've run into network issues as it will not pickup an ip address. I'll need to look into it further tomorrow.
 
Spent most of today trying to debug the problem and not getting very far with it. Decided it would be easier to debug down in the house rather than in the loft so I labelled all of the drives and cables, pulled the disks out and moved it downstairs. Once rebuilt, it was completely dead. I've tested the power supply on the old x79 system and that appears to be working fine. Took the working antec supply out of that and plugged it into this and got squat, won't even spin a fan. Tried testing it with no expansion cards and tried it with no ram same result. I suspect the motherboard has died.

The 790FX and phenom could go back in and get it working but a new board is definitely needed in order to accommodate the additional expansion card for 10Gbe. Dilemma is do I bite the bullet and go new and write off the 20GB of DDR3 I have laying around or find something used that will utilise that ram.
 
The budget side to this project just went out of the window. I have caved in and gone and bought the following after a long hard think.

i5 9600k
Asus Prime Z390-A
16GB Teamgroup vulcan 3000C16

Was looking to go ryzen with a 1600 or 2600, however, looking at the pcie lanes available, I'd not have enough to run the nvs310 gpu, raid card and 10Gbe NIC. That meant going with something with an igp. With ryzen there are only the 2200g & 2400g which are ok, but I wanted to move up to a hex core. That left me looking at intel, even though ryzen has 8 more pcie lanes, i'd have to tie 16 of them up with a gpu meaning i'd be 8 lanes worse off than going intel. Considered the i5-8400 as that's the cheapest 6 core they do, but considering the £40 difference between that and the 9600k oem chip I felt the 9th gen part was more sensible considering it's soldered and its a K sku. The challenge now will be to get the cooler to fit as I have an original prolimatech megahalems which only came with the 775/1366 mounting parts. I bought the amd retention kit separately back in 2009, but the 115x kit seems to be very hard to find these days.
 
Good point on that gpu, didn't realise they offered anything with an x1 slot.

That said, the price difference excluding getting one of those x1 gpus is £84, add £40-50 on the ryzen build cost and it's between £30 & £40 difference. I don't think it's all that worthwhile trying to change it now.
 
Back
Top Bottom