Linux 2GB filesize limit

Craig321 · 12 Jun 2006 at 18:28

Hi,

Is there any way to get round the Linux 2GB file size limit?

I'd like to mirror Vista onto my server, but when it gets past 2GB downloading it the file disappears and it gives access denied when trying to go to the file in a web browser.

Is there any way I can get round this limit please?

Thanks,
Craig.

Berserker · 12 Jun 2006 at 20:01

Linux doesn't have a 2GB file size limit (unless it's really old). It's the download tool you're using that does, usually because someone only allocated 31 bits to the file size (very common problem).

Let us know which tool you're using, and someone might be able to find an alternative.

Craig321 · 13 Jun 2006 at 14:46

Berserker said:
Linux doesn't have a 2GB file size limit (unless it's really old). It's the download tool you're using that does, usually because someone only allocated 31 bits to the file size (very common problem).

Let us know which tool you're using, and someone might be able to find an alternative.

I've tried curl -o, wget, lynx and all 3 have messed up past 2GB

Craig.

riddlermarc · 13 Jun 2006 at 15:27

is the following any good..? http://www.fedoraforum.org/forum/showthread.php?s=&threadid=2411

Moredhel · 13 Jun 2006 at 15:28

What filesystem are you using? As it's the filesystem that would have this kind of limit, and most modern ones don't.

leezer3 · 13 Jun 2006 at 16:36

AFAIK, its some sort of a bug in a library used by both curl & wget. I would assume Lynx is using the same library also.
I can only find references to it from 2004/ 2005, so my first point of call would be a complete update & compilation of the latest kernel.

The filesystem could also be a problem, but only really if you're using something odd.

-Leezer-

Caged · 13 Jun 2006 at 20:12

FYI, I used wget on Windows to get my Vista DVD, and it worked perfectly. I'd assume that the *nix wget would be fine also.

Burbleflop · 13 Jun 2006 at 20:21

Caged said:
FYI, I used wget on Windows to get my Vista DVD, and it worked perfectly. I'd assume that the *nix wget would be fine also.

It depends which version of wget is being used. Version 1.10 and above (I think) support files > 2Gb. Older versions don't.

Berserker · 13 Jun 2006 at 20:31

Correct. The one most people have is the older wget that doesn't cope. Newer version has been fine for me (actually, I've been using a hacked one, but if the official version now supports > 2GB, I'll switch back).

Burbleflop · 13 Jun 2006 at 20:34

I've been using a hacked version too but couldn't find a hacked version for OS X so compiled the latest from source and found that it didn't die when I downloaded a 4Gb DVD image.

Berserker · 13 Jun 2006 at 20:39

What version would that be? My Gentoo box has 1.10.2. Here's hoping it's a good'un as I'm about to go grab Vista myself.

Burbleflop · 13 Jun 2006 at 20:42

1.10.2

Moredhel · 13 Jun 2006 at 20:47

Hmm, I'm on 1.8.2 here, vista came down fine. :confused:

Burbleflop · 13 Jun 2006 at 20:50

Moredhel said:
Hmm, I'm on 1.8.2 here, vista came down fine.

That's surprising, 1.8.2 is over 3 years old.

leezer3 · 13 Jun 2006 at 21:10

See my post above

Its not specifically wget thats the problem, rather a low-level library somewhere. I'd hazard a guess that this is something to do with kernel revisions; OP, could we have your kernel version please?
(2.4 series bug?)

-Leezer-

Burbleflop · 13 Jun 2006 at 21:14

wget did have a known problem with files > 2Gb, nothing to do with libraries.

A quick google says that wget versions < 1.10 released after 13/11/2004 (so that'd be 1.9.1) can't cope with files > 2Gb. Seems that versions prior to that don't have a problem.

Berserker · 13 Jun 2006 at 21:24

Yes, I can back up this information. It is most definitely not a kernel or library problem in the case of wget. It's just a plain old bug in wget itself.

Craig321 · 13 Jun 2006 at 21:50

Wget 1.10.2 - can't remember the exact version of Red hat we're using, but it's a recent version.

riddlermarc said:
is the following any good..? http://www.fedoraforum.org/forum/showthread.php?s=&threadid=2411

The seg size was limit to what looked like 2GB on my normal account, so i logged into root and tried from that and all was unlimited but it still wouldn't let me download more than 2GB

leezer3 said:
AFAIK, its some sort of a bug in a library used by both curl & wget. I would assume Lynx is using the same library also.
I can only find references to it from 2004/ 2005, so my first point of call would be a complete update & compilation of the latest kernel.

The filesystem could also be a problem, but only really if you're using something odd.

-Leezer-

I'll check with my server admin if we're on the latest kernel.

Berserker · 13 Jun 2006 at 21:56

We've got a Red Hat Enterprise Linux 3.0 server with a copy of the 32-bit Vista install. Admittedly I patched the kernel to 2.6.16(ish) on that one, so it's not the original kernel.

Anything running 2.4 or newer with a decent file system (e.g. ext3) should cope fine.

'uname -r' should get the kernel version.

Craig321 · 13 Jun 2006 at 22:03

Berserker said:
We've got a Red Hat Enterprise Linux 3.0 server with a copy of the 32-bit Vista install. Admittedly I patched the kernel to 2.6.16(ish) on that one, so it's not the original kernel.

Anything running 2.4 or newer with a decent file system (e.g. ext3) should cope fine.

'uname -r' should get the kernel version.

2.6.9-34.0.1.ELsmp