File transfer speeds seem very slow.

RimBlock

RimBlock

RimBlock

Associate
Joined
10 Nov 2004
Posts
2,237
Location
Expat in Singapore
Hi,

I have my Linux (CentOS 5.5) NAS set-up but it seems to be very slow when I am transferring a lot of data from my Win7 PC to it.

NAS;
Intel C2D E6400
4GB Ram
320GB 5400rpm HDD (boot).
1TB 7200rpm HDD (data1).
5* 1.5TB WD-CG in software raid5 logical volume (Raid1) - shared with Samba.
2* Intel 1000/Pro GT PCI network cards (both connected with seperate ip addresses)

Switch;
Linksys WRT610N (Gbit switch)

PC;
Intel C2Q Q9450
8GB ram
SSD boot drive
1.5TB 7200rpm hdd (data1)
Intel 1000/Pro GT PCI network card

Actions (both happening at the same time);

Copying from NAS data1 -> NAS Raid1 (600GB data mostly video) - 8 hours and only 1/3 done
Copying from PC data1 -> NAS Raid1 (1TB data mostly video files) - 7 hours and ??? done

The PC -> NAS Raid1 copy peaked at 10MB/s but is averaging 6MB/s. I have no idea how fast the NAS to NAS copy is going as I am using rsync from the command line.

The only other real network traffic would be a bit of internet traffic through the same switch from another machine but this would max out at around 200Mb/s (50MB/s) or so.

Not sure where the bottleneck is (switch, network load, software raid, raid5).

Would appreciate someone with more experience than myself to point me in the right direction for the best ways to troubleshoot (with tool names etc if relevant). I am more than happy at the Linux command line but am not so aware of the GUI tools available or doing this SA level troubleshooting.

Thanks
RB
 
Ssh to the NAS and run the 'top' command. It's like task managerbut for linux.

See if there is much cpu time in the 'id' state, meaning idle. If the cpu is running flat out, that would show it.

What bus are the drives and nic connected to in this server of yours?
 
Well I used to run a 3 drive raid 5 on a pentium 2, and that could saturate a 100mbit link in either reads or writes... So it's not all that shabby!
 
Been running the system monitor on both the Windows 7 and Linux box with roughly there results.

PC (Win7)
Mem 1.5/8GB used
CPU: mot more than 20% on any of the 4 cores
Network: bouncing between 20%-40% for a couple of minutes then a spike to 50%-60% for around 10 seconds before falling back to vibrating between 20%40%.

Linux box
Both cores below 20%
Mem 1.5/4GB
Network: around 40MB/s but bouncing all over the place.

Time / volume of data gives 11.1MB/s (Windows was showing 10.0)

Stopping all torrent network traffic (going through the same switch) increased it to 10.6MB/s

I will now try resetting the switch and both machines and transferring a file.

All network cards are PCI (Linux*2 and PC*1).

The switch is a netgear GS605 (moved the other one to another room and forgot) but is still listed as a 5 port Gigabit switch.

Cheers
RB
 
Here is the results.

Only two machines connected to the switch and both machine and switch rebooted.

Small files
10gb (777 files 125 folders)
Starting speed:49MB/s
End speed: 18.5->19MB/s

Large files
7.8GB (20 files, 10 folders)
Starting speed: 70MB/s
Ending speed 20.5MB/s

Single large file
10.5GB (1 file)
Starting speed: 70MB/s
Ending speed 23MB/s

The speed drops to around 20MB/s 1/3 to 1/2 of the way through the transfer.

I can only get around 6.5MB/s from my DLink DNS-323 NAS box which is a pain. Would explain the stuttering when playing HD movies from it though.

I am leaning towards an IP address conflict or the switch filling up its IP tables.

Do the speeds above look reasonable for software raid 5 with 5*5400rpm drives.

Cheers
RB
 
A GS605 should be fine for 3 hosts.

Software RAID, which isn't quick at the best of times, plus 5400 drives for RAID5 is most definitely going to be bottlenecked all over the place simply because of IOPs.
Maybe with a proper controller you might have more luck, but RAID needs IOPs.

Perhaps a better solution for you would be RAID10, since it doesn't incur any IOP overhead while still giving a worthwhile speed increase. You will lose a bit of capacity but large disks are cheap these days.

Also keep in mind that most gigabit adapters do not achieve full speed.
Unless you're spending ~£20 at least on the network card, it'll probably only achieve 400 - 600.
The Intel gigabit adapters are good value for money if this is what you're after.
 
A variety of factors could be at play here:
Firstly, you said that network utilisation is only reported at being between 20 and 60% - If you're only getting around 6MB/s this implies the link is only 100Mb/s, rather that the full 1000 you should be getting. Check ifconfig and the settings on the PC doing the transfers.

After that as mentioned above, software RAID5 write speeds are usually pretty poor. your read speeds look pretty similar at the moment, but once you have the link running at proper Gigabit speeds it could definitely start limiting you. Going RAID10 instead is a solid choice.

Next thing it could be is that (as I recall) The linux implementation of SMB (protocol used for windows file sharing) is pretty slow. See what performance is like if you set up an FTP server or (with some help from google), ISCSI.

You can also use jPerf or iPerf to test the raw TCP transfer speeds between two computers without disk drives or other protocols getting in the way. Knock the window size up from 8KB to 54KB or so however.
Once you start getting 300-400Mb/s in iperf, you can move on to tweaking things. Bump up the recive windows and buffer sizes as shown here - http://fasterdata.es.net/TCP-tuning/linux.html . disabling Flow Control on your network cards can also help a bit.
 
Last edited:
Yamahahahahaha said:
A GS605 should be fine for 3 hosts.
Click to expand...

Thanks for the confirmation.

Yamahahahahaha said:
Software RAID, which isn't quick at the best of times, plus 5400 drives for RAID5 is most definitely going to be bottlenecked all over the place simply because of IOPs.
Maybe with a proper controller you might have more luck, but RAID needs IOPs.
Click to expand...

Have a thread in networking about the whole home server fun I have been having and the issues with finding a 'cheap' raid controller which is a raid controller (rather than a HBA even though it was advertised as a raid controller or a controller which can only handle 1TB drives). Still trying to find out if the Adaptec 4805 can handle > 1TB drives and greater than 2TB arrays. I can get one for < 180 USD. Prec6is have also been mentioned but there seem to be a number of models at wildly different prices second hand. TBH unless it says that it will handle my WD-CG 1.5TB drives or someone can verify and I can get fro under USD200 then it is really out of scope for a home network for me.

Yamahahahahaha said:
Perhaps a better solution for you would be RAID10, since it doesn't incur any IOP overhead while still giving a worthwhile speed increase. You will lose a bit of capacity but large disks are cheap these days.
Click to expand...

So I would need to move up to 6 drives and would be out of slots on the motherboard. Getting an adaptor to give more..... see comments about getting raid cards and ending up with the wrong thing .... twice above... ;). I have a number of spare drives so dropping to a 4 drive raid 10 is not preferred. The drives are also connected via an Icydock 5 bay internal hotswap unit.

Yamahahahahaha said:
Also keep in mind that most gigabit adapters do not achieve full speed.
Unless you're spending ~£20 at least on the network card, it'll probably only achieve 400 - 600.
The Intel gigabit adapters are good value for money if this is what you're after.
Click to expand...

I have 3 Intel GT 1000/Pro adapters (two in the server and one in the PC).

Zarf said:
A variety of factors could be at play here:
Firstly, you said that network utilisation is only reported at being between 20 and 60% - If you're only getting around 6MB/s this implies the link is only 100Mb/s, rather that the full 1000 you should be getting. Check ifconfig and the settings on the PC doing the transfers.
Click to expand...

Yes, only 6MB/s from my Dlink to the PC which does indicate 100Mbit although I though the DLink 323 had a GBit network interface.

My PC to the NAS server gets around 20MB/s as per the tests above.

Zarf said:
After that as mentioned above, software RAID5 write speeds are usually pretty poor. your read speeds look pretty similar at the moment, but once you have the link running at proper Gigabit speeds it could definitely start limiting you. Going RAID10 instead is a solid choice.
Click to expand...

Two recommendations make it hard to resist :D

Zarf said:
Next thing it could be is that (as I recall) The linux implementation of SMB (protocol used for windows file sharing) is pretty slow. See what performance is like if you set up an FTP server or (with some help from google), ISCSI.
Click to expand...

Ahh, two main tasks for the server.... to receive new files from either my Win7 or DLink 323 (Linux) box and to allow my WDHDLiveTV boxes to pull me3dia from it. The server will have possibly have upto 6 HD streams coming from it at any one time. I will also set it up as a local DNS, DHCP and possibly web server.

Zarf said:
You can also use jPerf or iPerf to test the raw TCP transfer speeds between two computers without disk drives or other protocols getting in the way. Knock the window size up from 8KB to 54KB or so however.
Once you start getting 300-400Mb/s in iperf, you can move on to tweaking things. Bump up the recive windows and buffer sizes as shown here - http://fasterdata.es.net/TCP-tuning/linux.html . disabling Flow Control on your network cards can also help a bit.
Click to expand...

Great, I will take a look.

iaind said:
Why is the nas multi homed?
Click to expand...

Personal preference with built in future proofing.

Thread about the home network I am building is here for more background.

Any tools available for benchmarking the raid5 array on the Linux box to see just how fast it is ?.

Many thanks for all the suggesting and info.

RB
 
Ok, I have just purchased a 2nd hand PERC6i so we will see how this works.

Still change to raid10 or keep at raid5 on the PERC ?

Thanks
RB
 
PERC6/i cards can take up to 16 attached drives.
You could try making a RAID5 on the controller, and if that's unsatisfactory, set up a RAID10 in parallel and move stuff across.
 
Strange using a wrt610 here and when i transfer any file from my wife's pc to my own, just a direct drag drop through the network, the average speed I am getting is 70MB/sec samsung f1 drive to samsung f2 drive, or other way round, it is just as fast to pull a file from her pc over the network as it is to transfer from one drive to another within my own machine.

Tested with small file and severla 700MB files.
 
I could not stream from the drives as whilst playing a 4GB mp4 movie it was hanging and then trying to catch up after around 20 minutes in to the film. A single drive light was on during the time the movie froze.

When streaming the movie from my little D-Link I got the same issue but after only 5-10 minutes in to the film.

Playing from my PC resulted in no issues so the movie file is fine.

I decided to take the raid10 advice and backed up then blew away the raid5 set.

Using mdadm I have created a raid 10 set.... but ..... on checking the status in /proc it reports it will take 33+ hours to create and speed is only 21MB/s. On the guide I followed the guy was using 7200K drives but got over 152MB/s.

To create:
mdadm --create /dev/md0 -v --raid-devices=4 --level=raid10 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1

/proc/mdstat:
Personalities : [raid6] [raid5] [raid4] [raid10]
md0 : active raid10 sdf1[3] sde1[2] sdc1[1] sdb1[0]
- - - - 2930271872 blocks 64K chunks 2 near-copies [4/4] [UUUU]
- - - - [>....................] resync = 0.0% (2676864/2930271872) finish=2129.4min speed=22912K/sec

The drive with the continual light during the raid5 slowdown is not part of the raid10 array (just by luck rather than choice).

I am tempted to kill this one and try a raid0 to see what speed that reports.

I am also looking at building hdparm..... seems I have to build from source code :(. Grabbing the C compiler now.

RB
 
Last edited:
Stranger and stranger.....

hdparm installed surprisingly easily.

hdparm -I /dev/.....

sda - boot drive - recognised
sdb - 1.5TB WDCG - HDIO_DRIVE_CMD(identify) failed: No such device
sdc - 1.5TB WDCG - recognised
sdd - 1.5TB WDCG - recognised
sde - 1.5TB WDCG - recognised
sdf - 1.5TB WDCG - recognised
sdg - 1TB drive on adaptec array controller - recognised (sort of)
sdh - 400GB drive on adaptec array controller - recognised (sort of)

My concern is with sdb.

Pullying the drives from the Icydock (5 bay unit), I can work out which drives are which but on putting them back again, a couple are reporting no such file.....

[root@nas hdparm-9.36]# hdparm -I /dev/sdd
/dev/sdd: No such file or directory

I am now a bit concerned about the Icydock :(
 
Looks like it was trying to rebuild the raid array too; normally that runs at a low priority - when mine rebuilt (I accidentally unplugged a drive for a day..), I couldn't "tell" it was rebuilding when accessing or copying files.

It does sound as if your hard drive subsystem is poorly though.
 
Well a little bit more of an update.

Creating a raid 10 array was getting around 22MB/s
Creating a raid 5 array was getting around 89MB/s

Then I noticed that the raid 5 array was missing a disk (it was specified in the create mdadm command). I added the disk and it started to rebuild again adding the disk in.

I noticed that one disk was again staying on while the others weren't every now and then. As the same time the speeds would drip from 89MB/s to 22MB/s. I then pulled all the 5 array disks and attached them to the Adaptec controller to check the write cache status (on or off) as I thought this one drive may have had it's cache turned off making it potentially slower than the other drives. Unfortunately I was wrong. All drives had the cache turned on. Unfortunately the OS seems to have put GRUB on one of the array disks. The off shoot being that the OS is now not booting as the right disk is not in the right slot. Add to that the fact that my CentOS 5.5 DVD is now reporting errors and I am now having to burn a new one.

One thing I did notice in the install screens before the DVD failed was that one of the raid drives was listed as being formatted with NTFS. I am wondering if this caused issues. I have now reformatted it.

Time for another install attempt.

RB
 
You must log in or register to reply here.
Back
Top Bottom