Linux 2GB filesize limit

Berserker · 13 Jun 2006 at 22:10

That'll be at least Red Hat Enterprise Linux 4.0 then, since that's the first one to use 2.6 kernels (without patching them manually). I don't see how that could be causing problems other than the one you already encountered (quota/ulimit), unless you're using some exotic filesystem type (like FAT32 :eek:

).

NickK · 14 Jun 2006 at 19:04

SuSE 9.3 + updated GCC, kernel recompile 2.6.11+ IIRC.

Downloaded the current stable wget build from sourceforge and that worked like a dream with the 4GB 64bit Vista download - including coping with an automated restart after disconnect.

R4z0r · 14 Jun 2006 at 19:55

Are you writing to a SAMBA share by any chance? If so, there is a 2GB limit (Last time I looked into it anyway)...

Your best bet would be to transfer the file using NFS, SCP, FTP, etc.

Craig321 · 14 Jun 2006 at 20:21

R4z0r said:
Are you writing to a SAMBA share by any chance? If so, there is a 2GB limit (Last time I looked into it anyway)...

Your best bet would be to transfer the file using NFS, SCP, FTP, etc.

Nope, not writing to a samba share.

How would I transfer from an FTP:// URL using SCP?

Thanks,
Craig.

R4z0r · 14 Jun 2006 at 21:09

Craig321 said:
How would I transfer from an FTP:// URL using SCP?

Thanks,
Craig.

You'd need to mess about with SSH tunneling, no point for what you're doing. It doesn't really matter anyway as you're not copying to an SMB share anyway !!

Can you post the output of "mount" and also advise the directory you are downloading to. Well, it doesn't need to be the exact directory but enough so we know which partition!

Also, are you sure the partition you are downloading to is not running out of space? Check with "df -h".

To rule out some possible problems, have you tried downloading from another source - Maybe a Linux DVD ISO to see if you still get the 2GB limit?

Craig321 · 14 Jun 2006 at 22:43

R4z0r said:
You'd need to mess about with SSH tunneling, no point for what you're doing. It doesn't really matter anyway as you're not copying to an SMB share anyway !!

Can you post the output of "mount" and also advise the directory you are downloading to. Well, it doesn't need to be the exact directory but enough so we know which partition!

Also, are you sure the partition you are downloading to is not running out of space? Check with "df -h".

To rule out some possible problems, have you tried downloading from another source - Maybe a Linux DVD ISO to see if you still get the 2GB limit?

Mount:

/dev/hda3 on / type ext3 (rw,usrquota)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/hda1 on /boot type ext3 (rw)
/dev/hdd3 on /old type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/usr/tmpDSK on /tmp type ext3 (rw,noexec,nosuid,loop=/dev/loop0)
/tmp on /var/tmp type none (rw,noexec,nosuid,bind)

I think im using either hda3 or hdd3 - I'm using the disk that user files are stored in which from what I can tell is hda3 but I'm not 100% certain.

I'm going to test a Linux DVD ISO now, although I don't think it'll work as the other day I was taring some stuff up and it broke when the tar got >2GB.

-- Edit --
Looking at it closer now I've been trying to download it onto hda3.

-- Edit 2 --
Also, I did df -h and the server has loads of space left.

-- Edit 3 --
Right, tested it on a SuSE DVD ISO (3.5GB), luckily I got a nice mirror which was located in the same country as the server so it was nice and fast. It did the same thing, after 2GB the file size dissappeared from the directory listing but the file appeared to still be there and the download was still going, but when I tried accessing the file from my browser I just got access denied

Thanks,
Craig.

Augmented · 14 Jun 2006 at 23:58

Craig321 said:
It did the same thing, after 2GB the file size dissappeared from the directory listing but the file appeared to still be there and the download was still going, but when I tried accessing the file from my browser I just got access denied

So you're attempting to download the file from Apache over HTTP?
Check the version of Apache you're serving the file with. Apache <2.2 does not support files larger than 2GB.

http://httpd.apache.org/docs/2.2/new_features_2_2.html (See Large File Support),

Craig321 · 15 Jun 2006 at 02:02

Augmented said:
So you're attempting to download the file from Apache over HTTP?
Check the version of Apache you're serving the file with. Apache <2.2 does not support files larger than 2GB.

http://httpd.apache.org/docs/2.2/new_features_2_2.html (See Large File Support),

I'm not 100% sure what you mean by attempting to download the file from Apache? I'm just trying to get a file from a ftp or http address using wget.

Also, if the version of apache we had didn't support >2GB wouldn't the file still show up in FTP?

Thx
Craig.

Burbleflop · 15 Jun 2006 at 07:41

Craig321 said:
I'm not 100% sure what you mean by attempting to download the file from Apache? I'm just trying to get a file from a ftp or http address using wget.

Also, if the version of apache we had didn't support >2GB wouldn't the file still show up in FTP?

Thx
Craig.

It is nothing to do with the version of Apache you have, it is to do with the version of Apache that the site you're downloading from has. But, if you're getting the same problem with FTP then I think we can ignore Apache.

What version of wget have you got on that box?

Berserker · 15 Jun 2006 at 11:51

wget --version

In case you're wondering.

However, I suspect you might be downloading Vista to share with others. If that's the case, then don't. Microsoft are clamping down on unofficial download sites. This being the case, you'd be far better off just downloading the thing from wherever you want it to end up.

Craig321 · 15 Jun 2006 at 12:53

Well, my original plan was to host vista for a bit for other people on OcUK, but not much point now as most people now have it.

Anyway, I'd still like to get it sorted out for the future

WGET version 1.10.2, oh also, I've tried mget as well in FTP in the shell.

Berserker · 15 Jun 2006 at 13:02

That version of wget is compatible with files > 2GB. We must be missing something that you're doing to 'break' it. :confused:

Craig321 · 15 Jun 2006 at 18:10

Ok, here's what I do:

Login to shell using my account / the root account (normal accounts seem to have a limit on filesize according to ulimit so I use the root account).

Type in wget http://address.com/directory/file.extension - this starts the download and downloads it into whatever folder i'm currently in.

When the download reaches 2GB I check it through Mozilla but I cannot see the filesize in the directory listing (I was able to see the filesize up until 2GB). So I wait for the download to finish and then try and download the file from the server but get access denied.

I check through FTP if the file's there but cannot see the file at all in FTP.

Craig.

adamofgreyskull · 15 Jun 2006 at 22:08

Craig321 said:
Ok, here's what I do:

Login to shell using my account / the root account (normal accounts seem to have a limit on filesize according to ulimit so I use the root account).

<snip>

Can you download it to your local machine? Or any machine? If you could find someone who wanted vista and get them to try to download it too?

However, I suspect you might be downloading Vista to share with others. If that's the case, then don't. Microsoft are clamping down on unofficial download sites. This being the case, you'd be far better off just downloading the thing from wherever you want it to end up

Why download it anyway? What's the big fuss, apart from the obvious rush people seem to get from being "naughty".

If I'm missing something and windows really is as awesome as people make out, why not download it to an IIS server

Craig321 · 16 Jun 2006 at 03:28

"naughty"? Beta 2 is public right now and free for download.

Anyway, I don't want to mirror Vista anymore, just wanna get this 2GB problem sorted for future stuff.

Can't download to an IIS server as the only server I have access to is a Linux one which is based in Texas.

Craig.

Una · 16 Jun 2006 at 20:35

Have you somehow managed to set ulimits by accident?

'ulimit -f' Try that in a shell for more info.

Craig321 · 16 Jun 2006 at 23:03

Hi,

Checked that, there seems to be a limit on normal user accounts but it's all unlimited on the root account.

Craig.

Una · 16 Jun 2006 at 23:32

So you made sure you were running wget as root? (stupid tbh) Otherwise if you have ulimits on your user accounts and exec wget as that your going to be limited.

adamofgreyskull · 17 Jun 2006 at 00:11

Craig321 said:
"naughty"? Beta 2 is public right now and free for download
Craig.

Windows doesn't make you feel like you need a shower? No? Just me?

Craig321 · 17 Jun 2006 at 01:01

Una said:
So you made sure you were running wget as root? (stupid tbh) Otherwise if you have ulimits on your user accounts and exec wget as that your going to be limited.

Well I've been running wget to try sort this prob since I found the normal user accounts have a ulimit of 2GB.

adamofgreyskull said:
Windows doesn't make you feel like you need a shower? No? Just me?

Just you