Rest of the kit arrived whilst I was in work today.
Home Server + 10G ethernet
- Thread starter Kei
- Start date
Job done. Second backplane was an extremely tight fit and took some serious persuasion. Fans are a little loud but not a patch on a real server. It now works correctly and the raid card is finally seeing the disks connected to it. (inc those on the backplanes after I realised I'd plugged the wrong ports in)
The back could possibly be a bit tidier, but it could have been far worse. (so it'll do as it shouldn't impede airflow or affect functionality)
The back could possibly be a bit tidier, but it could have been far worse. (so it'll do as it shouldn't impede airflow or affect functionality)
Went through the ordeal of trying to install ubuntu server onto the raid 10 array. First off I couldn't find it at all, just the LSI card and the raid 5 array. Tried setting the onboard SB750 sata back to AHCI and IDE modes then gave up and pulled the LSI card out. Drive list now shows empty. Gave up trying to use the gui method of installing and opted to try the CLI. Some moderate success was had here using parted and mdadm as the devices were present. (/dev/sda, sdb, sdc, sdd) Managed to fudge my way through creating a software raid using mdadm as device /dev/md0. Go back to installer and hey presto the device is listed as a 999GB disk ready to partition. Try to set it going using the partition tool and it fails to write changes to the disk. (even after I waited for the synchronisation to finish) 6 hours last night wasted.
Going to try and run the motherboard "fakeraid" again and see if i can find it using the CLI. (Something I didn't try) I can't work out how to get linux software raid to work. I can't use the LSI as I don't have sufficient SAS ports to run 12 disks and an expander is quite an expensive way to get one extra port. Might just give up and resort to windows server instead.
Going to try and run the motherboard "fakeraid" again and see if i can find it using the CLI. (Something I didn't try) I can't work out how to get linux software raid to work. I can't use the LSI as I don't have sufficient SAS ports to run 12 disks and an expander is quite an expensive way to get one extra port. Might just give up and resort to windows server instead.
I've run into some difficulties trying to set it up as I can't persuade grub2 to boot an mdadm raid 10 array. I've been trying to sort it out for 3 days and have finally given up. I can see a few different choices available:-
1. Buy an hp sas expander card and use the LSI to run the raid 10 array in addition to the raid 5 array
2. Bin the whole raid 10 array idea and just use my spare ssd for the OS
3. Run windows server instead
The choice that seems most logical to me is #2 but buy the expander anyway as dumping the 4 OS disks out of the case would allow me to use all 12 for the storage array. (or more should I upgrade to a larger case)
1. Buy an hp sas expander card and use the LSI to run the raid 10 array in addition to the raid 5 array
2. Bin the whole raid 10 array idea and just use my spare ssd for the OS
3. Run windows server instead
The choice that seems most logical to me is #2 but buy the expander anyway as dumping the 4 OS disks out of the case would allow me to use all 12 for the storage array. (or more should I upgrade to a larger case)
After some considerable chaos trying to get things to work via ubuntu, I gave up and moved to openSUSE and things have been plain sailing from there. I binned the RAID 10 array and used the spare SSD for the OS. This has freed up 4 more drive bays so I have also gone a picked up an intel Res2sv240 sas expander card to increase my port capacity to 16 dual linked. I can expand that to 20 if i single link it, and adding a second expander to the other port would give me 40. I can also daisy chain expanders too but I somewhat doubt I'll ever need that much space. When testing the array it completely saturated the gigabit lan connection managing a sustained write speed of 109mb/s for large video files. Smaller files dropped down into the mid 50's. Read speeds seem to be nice and high at ~4-500mb/s. (this was using just 4 disks)
I also got remote desktop via vnc to work too.
I also dropped another 2 disks into the array this morning to expand the storage up to a shade under 10TB. Reconstructing the array is taking a very long time though. It's taken 12 hours to get to ~40%. The icybox backplanes look great when running
I also got remote desktop via vnc to work too.
I also dropped another 2 disks into the array this morning to expand the storage up to a shade under 10TB. Reconstructing the array is taking a very long time though. It's taken 12 hours to get to ~40%. The icybox backplanes look great when running
Last edited:
I'll be running mine 24/7 as its healthier for the disks not to be spinning up and down frequently. (and i'd rather they last well as they are a huge cost in this project) I've dropped the voltage to the cpu and lowered the peak clock speed as I don't need it, but i reckon power usage will still be quite high. It will need to be fine tuned to get it to an optimum level.
Installed plex and hoping to configure it so that I can use it to stream old recordings lifted off my humax pvr. It works perfectly to my panasonic plasma but not got it to talk with vlc via upnp yet. (a non issue really as i can play the files direct via the shared storage) Still need to get the FTP server side of things running though.
6 disks finally migrated and fully initialized. Performance is pretty good. Intel expander should be here soon, though I have no need for it yet as I don't have enough disks to require it.
Installed plex and hoping to configure it so that I can use it to stream old recordings lifted off my humax pvr. It works perfectly to my panasonic plasma but not got it to talk with vlc via upnp yet. (a non issue really as i can play the files direct via the shared storage) Still need to get the FTP server side of things running though.
6 disks finally migrated and fully initialized. Performance is pretty good. Intel expander should be here soon, though I have no need for it yet as I don't have enough disks to require it.
Tested out FTP today and that works perfectly too, uploading to it at ~8-10mb/s and downloading from at ~2-3mb/s. Need to tinker with the settings a bit to make sure it is secure.
Intel SAS expander arrived today. May well look into fitting it over the weekend.
Intel SAS expander arrived today. May well look into fitting it over the weekend.
The intel sas expander is now fitted and working perfectly. I was expecting to have to rebuild the array from scratch but the controller picked up the virtual drive as if nothing had changed and is working exactly as before. A minor headache came when I tried to fit the card as it requires a pcie 4x sized slot and my gigabyte board only has two 16x slots and three 1x slots. I've had to use the vertical mount on the case for it until I can come up with something suitable to help neaten the cabling back out. (considering a 1x to 16x riser card for low profile cards) I also crimped the connectors for the 38mm NMB-MAT fan to aid cooling as I had to loose the side panel fan to fit the expander.
Added the third breakout cable and adjusted the fan power connections so that they are a tad quieter. Also installed the proper nvidia drivers which has finally cured the fan on the quadro running at full tilt all the time. It's finally reasonably quiet now. Got 7 disks connected giving me 12TB, still got room for another 5. (I've put some spares in for now to simulate a full case so i can monitor temperatures)
Last edited:
Associate
- Joined
- 1 Oct 2014
- Posts
- 100
Nice job, looks well worth the effort.
Soldato
Shame you gave up on Ubuntu server so quickly. GRUB won't load off of a RAID volume other than RAID 1 which is why you couldn't get it to work. You have to replicate a small (200Mb) boot partition in RAID 1 over all the drives and then the remainder can be made into RAID 10 if you want to do it the way you were thinking (how many mirrors of a boot?? lol).
Best solution would have been to do the root install to your SSD then RAID 10 the remaining drives and mount them onto /home, having said that RAID 10 is a fair amount of overkill for a home solution (even though I use it myself) and you would probably find yourself wishing in a few months time you'd have gone RAID 5 (or in your case maybe 6).
Best solution would have been to do the root install to your SSD then RAID 10 the remaining drives and mount them onto /home, having said that RAID 10 is a fair amount of overkill for a home solution (even though I use it myself) and you would probably find yourself wishing in a few months time you'd have gone RAID 5 (or in your case maybe 6).
I got ubuntu working although, I'm glad i ditched server as I couldn't work entirely using the CLI, much rather have a GUI. Opensuse was a breath of fresh air compared with ubunutu, the yast configuration panel really is brilliant for sorting out most network functions. Best of all, it was fairly easy to install the LSi megaraid management software so that i can mange things from across the network. I've been using it for transcoding mkvs using handbrake rather than tying up my main pc.
I moved on from the RAID 10/1 ideas and just stuck with a single SSD for the OS and RAID 5 spanned across 7 disks (so far) for the storage array which is mounted into /media rather than /home. I have been wondering whether RAID 6 with a 64/128k strip size may have been a more logical choice though. (i went for 256k)
I moved on from the RAID 10/1 ideas and just stuck with a single SSD for the OS and RAID 5 spanned across 7 disks (so far) for the storage array which is mounted into /media rather than /home. I have been wondering whether RAID 6 with a 64/128k strip size may have been a more logical choice though. (i went for 256k)
Last edited:
A full network rebuild has been performed as my old tp link 1043nd router has been requiring a prod every couple of days as it would just freeze up and stop all network traffic. I decided that i was up for a challenge and went with a cisco 1921/K9 ISR, an hp 1910-8G managed switch and a D-link DAP-2695 wireless access point. Complete and total overkill, I know, but I got them at a good price and being enterprise kit they ought to be reliable. (which was my main consideration) 5GHz wifi certainly is a lot quicker than 2.4 and the broadcast range from the d-link seems to travel slightly further too.
Having lived with it for just over a week, I can say that it has behaving well. (current PPPoE session has been up for 4 days so far, which is better than the tp link would usually manage) I certainly get better performance from it too. Very pleased there. (used to get low-mid 74 down and high 17- low 18 up, ping is about the same)
Config wise, it took me about 4 hours to configure the router sufficiently that i could access the internet. It took me a few days more and some assistance of the helpful network gurus to sort out the necessary static nat and access lists to get the server to be accessible from the outside for ftp, ssh and teamspeak. Still not sure on steam games as I don't do multiplayer very often. I'm not sure if I've covered all bases on the deny list but I'm hopeful that it's close.
The basic config for running with a bt infinity vdsl2 modem for those that are interested.
Having lived with it for just over a week, I can say that it has behaving well. (current PPPoE session has been up for 4 days so far, which is better than the tp link would usually manage) I certainly get better performance from it too. Very pleased there. (used to get low-mid 74 down and high 17- low 18 up, ping is about the same)
Config wise, it took me about 4 hours to configure the router sufficiently that i could access the internet. It took me a few days more and some assistance of the helpful network gurus to sort out the necessary static nat and access lists to get the server to be accessible from the outside for ftp, ssh and teamspeak. Still not sure on steam games as I don't do multiplayer very often. I'm not sure if I've covered all bases on the deny list but I'm hopeful that it's close.
The basic config for running with a bt infinity vdsl2 modem for those that are interested.
Code:
!
! Last configuration change at 12:44:44 UTC Tue Nov 11 2014
version 15.2
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname Cisco-1921
!
boot-start-marker
warm-reboot
boot-end-marker
!
!
enable secret 5.
enable password 7
!
no aaa new-model
!
ip cef
!
!
!!!!!!DHCP server config
ip dhcp excluded-address 192.168.0.1
ip dhcp excluded-address 192.168.0.50 192.168.0.51
!
ip dhcp pool main_dhcp_pool
network 192.168.0.0 255.255.255.0
default-router 192.168.0.1
dns-server x.x.x.x
!
ip dhcp pool Kei-PC
host 192.168.0.2 255.255.255.0
client-identifier xxx
client-name Kei-PC
!
ip dhcp pool Humax
host 192.168.0.4 255.255.255.0
client-identifier xxx
client-name Humax
!
ip dhcp pool Kei-NAS
host 192.168.0.3 255.255.255.0
hardware-address xxx
client-name Kei-NAS
!
ip dhcp pool Server
host 192.168.0.100 255.255.255.0
hardware-address xxx
!
!
!
ip domain name WesNet
ip name-server 8.8.8.8
no ipv6 cef
multilink bundle-name authenticated
!
!
!
license udi pid CISCO1921/K9 sn xxx
!
!!!!!Archive every 7 days or on writing config
archive
path ftp:/xxx/Cisco-1921
write-memory
time-period 10080
!
!
!
!
!
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
!
!!!!!! Note that Ge0/0 has no IP address
interface GigabitEthernet0/0
no ip address
ip tcp adjust-mss 1452
duplex auto
speed auto
pppoe enable group global
pppoe-client dial-pool-number 1
!
!
!!!!!! This is the LAN side
interface GigabitEthernet0/1
ip address 192.168.0.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
ip tcp adjust-mss 1452
duplex auto
speed auto
no mop enabled
!
!!!!!! The ISP's given IP address will be configured via d1
interface Dialer1
ip address negotiated
ip access-group 101 in
no ip unreachables
ip mtu 1492
ip nat outside
ip virtual-reassembly in
encapsulation ppp
dialer pool 1
ppp chap hostname [email protected]
ppp chap password 7 xxx
ppp pap sent-username [email protected] password 7 xxx
!
ip forward-protocol nd
!
ip http server
no ip http secure-server
!
!!!!!! This is the dynamic PAT between Dialer1 (WAN) interface IP address
!!!!!! and local IP addresses within ACL 1
ip nat inside source list 1 interface Dialer1 overload
ip nat inside source static tcp 192.168.0.3 20 interface Dialer1 20
ip nat inside source static tcp 192.168.0.3 21 interface Dialer1 21
ip nat inside source static tcp 192.168.0.3 22 interface Dialer1 22
ip route 0.0.0.0 0.0.0.0 Dialer1
ip route 192.168.0.0 255.255.255.0 GigabitEthernet0/1
!
access-list 1 remark INTERNET-ACCESS
access-list 1 permit 192.168.0.0 0.0.0.255
!
access-list 101 remark DENY FAKE IPs
access-list 101 deny ip 192.168.0.0 0.0.255.255 any
access-list 101 deny ip 172.16.0.0 0.15.255.255 any
access-list 101 deny ip 10.0.0.0 0.255.255.255 any
access-list 101 deny ip 127.0.0.0 0.255.255.255 any
access-list 101 deny ip 255.0.0.0 0.255.255.255 any
access-list 101 deny ip 248.0.0.0 7.255.255.255 any
access-list 101 deny ip 224.0.0.0 7.255.255.255 any
!
access-list 101 remark DENY SPOOFING IPs
access-list 101 deny ip host 0.0.0.0 any
access-list 101 deny ip host 255.255.255.255 any
!
access-list 101 remark DENY VULNERABLE PORTS
access-list 101 deny tcp any any range 135 139 log-input
access-list 101 deny udp any any range 135 139 log-input
access-list 101 deny tcp any any eq 2000 log-input
access-list 101 deny tcp any any eq 2001 log-input
access-list 101 deny tcp any any eq 6000 log-input
access-list 101 deny tcp any any eq 6001 log-input
access-list 101 deny tcp any any range 5900 5910 log-input
access-list 101 deny tcp any any range 5800 5810 log-input
access-list 101 deny tcp any any eq finger log-input
!
access-list 101 remark DENY TRACEROUTE OUTSIDE
access-list 101 deny udp any any range 33400 34400 log-input
!
access-list 101 remark ALLOW ACCESS FOR SERVER FTP & SSH
access-list 101 permit tcp any any eq 20
access-list 101 permit tcp any any eq 21
access-list 101 permit tcp any any eq 22
!
access-list 101 permit udp any any eq ntp
!
access-list 101 permit icmp any any net-unreachable
access-list 101 permit icmp any any host-unreachable
access-list 101 permit icmp any any port-unreachable
access-list 101 permit icmp any any packet-too-big
access-list 101 permit icmp any any administratively-prohibited
access-list 101 permit icmp any any source-quench
access-list 101 permit icmp any any ttl-exceeded
access-list 101 permit icmp any any echo-reply
access-list 101 deny icmp any any
access-list 101 permit tcp any any established
access-list 101 permit udp any any
!
!
!
!
snmp-server community WesNet RO
snmp-server enable traps entity-sensor threshold
!
control-plane
!
!
!
line con 0
line aux 0
line 2
no activation-character
no exec
transport preferred none
transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
password 7
login
transport input all
!
scheduler allocate 20000 1000
!
end
Last edited:
Well with most of summer gone it seems the server barely even broke a sweat in the loft. Not seen higher than 50 degrees out of it so far, but then I'm hardly taxing the cpu. RAID array is still in a nice healthy state with all 7 WD Se disks behaving as they should. Considering upgrading my LSI 8888elp to something with newer standards that would allow me to attach an LTO tape drive to it and run LTFS. Presently I can run a tape drive but I can't run LTFS as the HBA doesn't support it. Backup software like symantec backup exec would work but it's not as convenient and I'm not even sure you can get it or an equivalent for opensuse.
No idea on the power consumption as yet, need to consult the bills and run a comparison as we're nearly a year down the road.
No idea on the power consumption as yet, need to consult the bills and run a comparison as we're nearly a year down the road.
Picked up a replacement GPU for the server. A nice tidy nvidia quadro NVS 310 with a measly 19.5W maximum power draw. The old FX3500 was a greedy card at 95W which was complete overkill for server purposes.
NVS310 by Kyle, on Flickr
I finally got VNC to work properly. It turned out that gnome 3 and even gnome classic DE's wouldn't play nicely with VNC due to hardware rendering being required. I've now switched to using XFCE which works perfectly. I've also switched the login screen to lightdm which is easier on the hardware too. Space is getting chomped up quite quickly due to the expansion of my plex library. I think my handbrake encode quality might be a tad high as some of the bluray encodes are well above 20GB a piece.
Home Server by Kyle, on Flickr
NVS310 by Kyle, on FlickrI finally got VNC to work properly. It turned out that gnome 3 and even gnome classic DE's wouldn't play nicely with VNC due to hardware rendering being required. I've now switched to using XFCE which works perfectly. I've also switched the login screen to lightdm which is easier on the hardware too. Space is getting chomped up quite quickly due to the expansion of my plex library. I think my handbrake encode quality might be a tad high as some of the bluray encodes are well above 20GB a piece.
Home Server by Kyle, on Flickr
Soldato
How are you finding the write speeds to the box from your PC? I'm turning the idea over of changing my RAID10 to a RAID5 and I'm slightly bothered about speeds dropping too low.
Essentially, it usually saturates the gigabit network link at 100MB/s, though I have seen it fall as low as 32 on occasion. I think my combination of stripe size and cluster size on the ext4 file system makes write speeds on smaller files slightly lower.