File transfer speeds seem very slow.

[RimBlock]

Hi,

I have my Linux (CentOS 5.5) NAS set-up but it seems to be very slow when I am transferring a lot of data from my Win7 PC to it.

NAS;
Intel C2D E6400
4GB Ram
320GB 5400rpm HDD (boot).
1TB 7200rpm HDD (data1).
5* 1.5TB WD-CG in software raid5 logical volume (Raid1) - shared with Samba.
2* Intel 1000/Pro GT PCI network cards (both connected with seperate ip addresses)

Switch;
Linksys WRT610N (Gbit switch)

PC;
Intel C2Q Q9450
8GB ram
SSD boot drive
1.5TB 7200rpm hdd (data1)
Intel 1000/Pro GT PCI network card

Actions (both happening at the same time);

Copying from NAS data1 -> NAS Raid1 (600GB data mostly video) - 8 hours and only 1/3 done
Copying from PC data1 -> NAS Raid1 (1TB data mostly video files) - 7 hours and ??? done

The PC -> NAS Raid1 copy peaked at 10MB/s but is averaging 6MB/s. I have no idea how fast the NAS to NAS copy is going as I am using rsync from the command line.

The only other real network traffic would be a bit of internet traffic through the same switch from another machine but this would max out at around 200Mb/s (50MB/s) or so.

Not sure where the bottleneck is (switch, network load, software raid, raid5).

Would appreciate someone with more experience than myself to point me in the right direction for the best ways to troubleshoot (with tool names etc if relevant). I am more than happy at the Linux command line but am not so aware of the GUI tools available or doing this SA level troubleshooting.

Thanks
RB

 

[RimBlock]

Been running the system monitor on both the Windows 7 and Linux box with roughly there results.

PC (Win7)
Mem 1.5/8GB used
CPU: mot more than 20% on any of the 4 cores
Network: bouncing between 20%-40% for a couple of minutes then a spike to 50%-60% for around 10 seconds before falling back to vibrating between 20%40%.

Linux box
Both cores below 20%
Mem 1.5/4GB
Network: around 40MB/s but bouncing all over the place.

Time / volume of data gives 11.1MB/s (Windows was showing 10.0)

Stopping all torrent network traffic (going through the same switch) increased it to 10.6MB/s

I will now try resetting the switch and both machines and transferring a file.

All network cards are PCI (Linux*2 and PC*1).

The switch is a netgear GS605 (moved the other one to another room and forgot) but is still listed as a 5 port Gigabit switch.

Cheers
RB

 

[RimBlock]

Here is the results.

Only two machines connected to the switch and both machine and switch rebooted.

Small files
10gb (777 files 125 folders)
Starting speed:49MB/s
End speed: 18.5->19MB/s

Large files
7.8GB (20 files, 10 folders)
Starting speed: 70MB/s
Ending speed 20.5MB/s

Single large file
10.5GB (1 file)
Starting speed: 70MB/s
Ending speed 23MB/s

The speed drops to around 20MB/s 1/3 to 1/2 of the way through the transfer.

I can only get around 6.5MB/s from my DLink DNS-323 NAS box which is a pain. Would explain the stuttering when playing HD movies from it though.

I am leaning towards an IP address conflict or the switch filling up its IP tables.

Do the speeds above look reasonable for software raid 5 with 5*5400rpm drives.

Cheers
RB

 

[RimBlock]

A GS605 should be fine for 3 hosts.

Thanks for the confirmation.

Software RAID, which isn't quick at the best of times, plus 5400 drives for RAID5 is most definitely going to be bottlenecked all over the place simply because of IOPs.
Maybe with a proper controller you might have more luck, but RAID needs IOPs.

Have a thread in networking about the whole home server fun I have been having and the issues with finding a 'cheap' raid controller which is a raid controller (rather than a HBA even though it was advertised as a raid controller or a controller which can only handle 1TB drives). Still trying to find out if the Adaptec 4805 can handle > 1TB drives and greater than 2TB arrays. I can get one for < 180 USD. Prec6is have also been mentioned but there seem to be a number of models at wildly different prices second hand. TBH unless it says that it will handle my WD-CG 1.5TB drives or someone can verify and I can get fro under USD200 then it is really out of scope for a home network for me.

Perhaps a better solution for you would be RAID10, since it doesn't incur any IOP overhead while still giving a worthwhile speed increase. You will lose a bit of capacity but large disks are cheap these days.

So I would need to move up to 6 drives and would be out of slots on the motherboard. Getting an adaptor to give more..... see comments about getting raid cards and ending up with the wrong thing .... twice above... . I have a number of spare drives so dropping to a 4 drive raid 10 is not preferred. The drives are also connected via an Icydock 5 bay internal hotswap unit.

Also keep in mind that most gigabit adapters do not achieve full speed.
Unless you're spending ~£20 at least on the network card, it'll probably only achieve 400 - 600.
The Intel gigabit adapters are good value for money if this is what you're after.

I have 3 Intel GT 1000/Pro adapters (two in the server and one in the PC).

A variety of factors could be at play here:
Firstly, you said that network utilisation is only reported at being between 20 and 60% - If you're only getting around 6MB/s this implies the link is only 100Mb/s, rather that the full 1000 you should be getting. Check ifconfig and the settings on the PC doing the transfers.

Yes, only 6MB/s from my Dlink to the PC which does indicate 100Mbit although I though the DLink 323 had a GBit network interface.

My PC to the NAS server gets around 20MB/s as per the tests above.

After that as mentioned above, software RAID5 write speeds are usually pretty poor. your read speeds look pretty similar at the moment, but once you have the link running at proper Gigabit speeds it could definitely start limiting you. Going RAID10 instead is a solid choice.

Two recommendations make it hard to resist

Next thing it could be is that (as I recall) The linux implementation of SMB (protocol used for windows file sharing) is pretty slow. See what performance is like if you set up an FTP server or (with some help from google), ISCSI.

Ahh, two main tasks for the server.... to receive new files from either my Win7 or DLink 323 (Linux) box and to allow my WDHDLiveTV boxes to pull me3dia from it. The server will have possibly have upto 6 HD streams coming from it at any one time. I will also set it up as a local DNS, DHCP and possibly web server.

You can also use jPerf or iPerf to test the raw TCP transfer speeds between two computers without disk drives or other protocols getting in the way. Knock the window size up from 8KB to 54KB or so however.
Once you start getting 300-400Mb/s in iperf, you can move on to tweaking things. Bump up the recive windows and buffer sizes as shown here - http://fasterdata.es.net/TCP-tuning/linux.html . disabling Flow Control on your network cards can also help a bit.

Great, I will take a look.

Why is the nas multi homed?

Personal preference with built in future proofing.

Thread about the home network I am building is here for more background.

Any tools available for benchmarking the raid5 array on the Linux box to see just how fast it is ?.

Many thanks for all the suggesting and info.

RB

 

[RimBlock]

Ok, I have just purchased a 2nd hand PERC6i so we will see how this works.

Still change to raid10 or keep at raid5 on the PERC ?

Thanks
RB

 

[RimBlock]

I could not stream from the drives as whilst playing a 4GB mp4 movie it was hanging and then trying to catch up after around 20 minutes in to the film. A single drive light was on during the time the movie froze.

When streaming the movie from my little D-Link I got the same issue but after only 5-10 minutes in to the film.

Playing from my PC resulted in no issues so the movie file is fine.

I decided to take the raid10 advice and backed up then blew away the raid5 set.

Using mdadm I have created a raid 10 set.... but ..... on checking the status in /proc it reports it will take 33+ hours to create and speed is only 21MB/s. On the guide I followed the guy was using 7200K drives but got over 152MB/s.

To create:
mdadm --create /dev/md0 -v --raid-devices=4 --level=raid10 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1

/proc/mdstat:
Personalities : [raid6] [raid5] [raid4] [raid10]
md0 : active raid10 sdf1[3] sde1[2] sdc1[1] sdb1[0]
- - - - 2930271872 blocks 64K chunks 2 near-copies [4/4] [UUUU]
- - - - [>....................] resync = 0.0% (2676864/2930271872) finish=2129.4min speed=22912K/sec

The drive with the continual light during the raid5 slowdown is not part of the raid10 array (just by luck rather than choice).

I am tempted to kill this one and try a raid0 to see what speed that reports.

I am also looking at building hdparm..... seems I have to build from source code . Grabbing the C compiler now.

RB

 

[RimBlock]

Stranger and stranger.....

hdparm installed surprisingly easily.

hdparm -I /dev/.....

sda - boot drive - recognised
sdb - 1.5TB WDCG - HDIO_DRIVE_CMD(identify) failed: No such device
sdc - 1.5TB WDCG - recognised
sdd - 1.5TB WDCG - recognised
sde - 1.5TB WDCG - recognised
sdf - 1.5TB WDCG - recognised
sdg - 1TB drive on adaptec array controller - recognised (sort of)
sdh - 400GB drive on adaptec array controller - recognised (sort of)

My concern is with sdb.

Pullying the drives from the Icydock (5 bay unit), I can work out which drives are which but on putting them back again, a couple are reporting no such file.....

[root@nas hdparm-9.36]# hdparm -I /dev/sdd
/dev/sdd: No such file or directory

I am now a bit concerned about the Icydock

 

[RimBlock]

Well a little bit more of an update.

Creating a raid 10 array was getting around 22MB/s
Creating a raid 5 array was getting around 89MB/s

Then I noticed that the raid 5 array was missing a disk (it was specified in the create mdadm command). I added the disk and it started to rebuild again adding the disk in.

I noticed that one disk was again staying on while the others weren't every now and then. As the same time the speeds would drip from 89MB/s to 22MB/s. I then pulled all the 5 array disks and attached them to the Adaptec controller to check the write cache status (on or off) as I thought this one drive may have had it's cache turned off making it potentially slower than the other drives. Unfortunately I was wrong. All drives had the cache turned on. Unfortunately the OS seems to have put GRUB on one of the array disks. The off shoot being that the OS is now not booting as the right disk is not in the right slot. Add to that the fact that my CentOS 5.5 DVD is now reporting errors and I am now having to burn a new one.

One thing I did notice in the install screens before the DVD failed was that one of the raid drives was listed as being formatted with NTFS. I am wondering if this caused issues. I have now reformatted it.

Time for another install attempt.

RB

 

[RimBlock]

Sorry for the confusion, I am building the array after the install. I just happened to notice during the install that one of the disks was listed as NTFS.

I have reinstalled, rebuilt the array (averaged 50-80MB/s according to /proc/mdstat). I have copied some data to it.

Upload speed: 20-30MB/s (copying data to the drive from another machine).
Download speed: 30-40MB/s (copying data to another machine from the other machine). Significantly up on 22MB/s previously.

..again I notice that one and somethings two drive lights are on a lot more than the others. Two of the drives are slightly older and seem to have different firmware but are the same model id for Western Digital Caviar Green (64MB). Maybe the older firmware is slightly slower.

Best / anyway to align the drive firmware ?

Thanks
RB

 

[RimBlock]

Ok, looks like the issue could be related to these drives being advance format drives and so having to align the partition to the drives geometry

Now to find an easy to understand guide for doing that.

RB

 

[RimBlock]

Right, the drive where the light kept on whilst the others were flickering had been tested with WD drive testing tool and has failed with too many bad sectors.

I have also used 'parted' to align to 4k sectors on the remaining 4 drives. The array is rebuilding now with an eta of around 260 minutes () but with a throughput of 90-93MB/s . Here is the link to the WD article on setting 4k blocks.

seems better but the proof will be with the copying to and from the drive.

Only 257minutes to go plus fs formatting time.

RB

 

[RimBlock]

Ok,

70->80MB/s download or upload from a PC from 2 raid0 vertex2 ssd's

Much happier. Hopefully when I get the faulty drive replaced and it rebuilds it will maintain the same speeds. Hopefully the Perc6 should come soon and then it all starts again.

RB

 

[RimBlock]

A few more stats for you.

I started copying my movies from the NAS back to a backup single drive on my workstation. 740GB. The speed dropped to 15MB/s within 5 minutes.

I then rebooted the workstation and the switch and it is now running at 53MB/s.

Results from HDParm;

[root@nas hdparm-9.36]# hdparm -tT /dev/md0

/dev/md0:
Timing cached reads: 6280 MB in 2.00 seconds = 3145.43 MB/sec
Timing buffered disk reads: 590 MB in 3.01 seconds = 196.23 MB/sec

Would be interested to know others benchmarks for like systems to know if the results are as expected / slow or, deep breath, actually quite good.

The Perc6 has arrived but I am still awaiting the cables. The motherboard in the NAS PC only has one PCI x16 slot and no on board video . New motherboard time. Have ordered a dual PCI-e x16 board as the Prec6 is a PCI-e x8 card so the PCI-e x1 slots are no good.

Thanks
RB