[TriLUG] question about 2 GB file size limitation in Linux

[Geoffrey Douglas Purdy]

The > 2 GB files are continuing to give me headaches.  The latest issue
is that Perl (5.6.0) dies when trying to open them with the following
error:
can't open hugefasta: File too large at ./test.pl line 11

It looks like a similar problem to my original question.  The following is
the gist of what I've seen solution-wise on the web:
1) Perl (on Linux) will not open files > 2 GB unless it is compiled with
the USE_LARGE_FILES directive.
2) Compiling with the USE_LARGE_FILES directive may break compatibility
with existing perl applications.

Do I need to recompile Perl in order to manipulate > 2 GB files?  If so,
am I likely to screw up my existing Perl applications?

(64-bit UNIX is starting to look quite attractive).

Apologies if this is a bit off-topic for the list - it's more of a
"Perl-on-Linux" question.

-Geoff




On Thu, 20 Dec 2001, Jeremy P wrote:

> On Thu, 20 Dec 2001, Geoffrey Douglas Purdy wrote:
>
> > I'm running into a problem analyzing large files ( > 2 GB ) under
> > Linux.  The analysis application fails with input files greater than 2
> > GB in size but works on input of 1.9 GB or less.  I hope to identify
> > whether the 2 GB limitation lies at the application level, in the ext2
> > filesystem, or the Linux kernel.
> >
> > I can perform copy, cat, tail operations on files as large as 5 GB
> > without problems so I suspect that the limitation isn't the 2.4 kernel
> > or ext2, however I've seen a "2 GB file size limit in Linux" mentioned
> > frequently.  Does this limit refer only to older kernels?
>
> The 2GB file limit is now only a problem in older versions of [g]libc, not
> the kernel or filesystem.  Make sure you're running the latest glibc (and
> understand the caveats with upgrading it).   But you're right that it's
> probably an application issue -- large files require a different API, so
> if your application doesn't know how to use them, it will fail.  It might
> not be too hard to modify it if you know a little C (and the app is open
> source).  If your app can work on a stdin/stdout basis that may help get
> around the problem too.
>
> A sligthly outdated document here describes some of the issues:
> 	http://www.suse.de/~aj/linux_lfs.html
>
> Read the glibc documetation for details on large file API.
>
> Hope this helps,
> Jeremy
>
> _______________________________________________
> TriLUG mailing list
> http://www.trilug.org/mailman/listinfo/trilug
>