[TriLUG] Files in a directory question

[Scott Lambdin]

I think creating a new file would be a better test.  If you access a file,
the FS might get lucky and it might be the first inode it checks.  But if
you touch a new file, the FS driver has to check all the filenames to make
sure the new name is unique.

On Sat, Feb 14, 2009 at 10:04 AM, Steve Litt <slitt at troubleshooters.com>wrote:

> On Friday 13 February 2009 02:23:43 pm Steve Kuekes wrote:
> > Fellow triluggers,
> >
> > Here's a question, how many files should I put in one directory on an
> > ext3 file system?  I've got an app that needs a bunch of jpg image
> > files.  Right now its ~8,000-10,000 but later it could be 2 or 3 times
> > that many.  So for performance reasons, should I split these files up
> > into multiple sub-directories.  I can split them up according to the
> > names and put just 100 or 1000 in each directory so all the files that
> > start with a,b,c,d are in one folder called abcd, etc.  or I can just
> > put them all in one folder.
> >
> > I know that theoretically it's probably a very large file number limit
> > before it breaks, but for practical performance reasons in accessing
> > files in the folder how many is too many?
>
> This is just one man's opinion.
>
> Unless you access this directory very, very rarely, you've already passed
> the
> point where you should make it into a directory tree rather than a single
> directory. If I remember correctly, 10,000 files yields a several second
> latency.
>
> I'd suggest you test this by writing a perl or ruby program that creates
> 10,000 different-named small files. Then cat one of the files, and see how
> long it takes to cat the thing. To give you an idea of how fast a file can
> load when unencumbered by massive directories, my UMENU program, which has
> a
> separate file for each submenu, displays before your keypressing finger has
> left the key, even on files that aren't cached. On a human scale, it's
> instant.
>
> HTH
>
> SteveT
>
> Steve Litt
> Recession Relief Package
> http://www.recession-relief.US <http://www.recession-relief.us/>
>
> --
> TriLUG mailing list        : http://www.trilug.org/mailman/listinfo/trilug
> TriLUG FAQ  : http://www.trilug.org/wiki/Frequently_Asked_Questions
>



-- 
There's a box?