[TriLUG] bash help

[Steve Litt]

On Thursday 28 October 2004 10:11 am, rwshep2000.2725323 at bloglines.com wrote:
> Hi,
>
> I have a server with a shared repository for files.  I plan to devote
> 70GB of an 80GB HD (a single data partition) to the files.  The files are
> uploaded and placed in the repository via a web application.  Here is what
> I'd like to accomplish:
>
> When directory size exceeds 70GB, delete files,
> First-In-First-Out, until the repository is pared back to 70GB.
>
> The best
> case scenario would be to pare back the files each time a new file is
> added. However, I am hoping to do this without adding web application
> logic, which could cause additional latency for the user.  Although it
> risks possibly exceeding the size limit, I am thinking of using a bash
> script scheduled with cron. To ensure against exceeding the limit, I'm
> leaving 10GB of the 80GB as buffer. I know this is imperfect but my humble
> intellect can't think of another approach.
>
>
> So I'm looking for comments on two things:
>
> 1.  How to make a bash script
> look at total directory size, 

df

> then proceed to delete files FIFO until a 
> target size is reached;

Loop: find the oldest, delete it, check disk space

Personally, I'd write this thing in Perl
>
> 2.  Whether there is a better alternative than putting
> this script on cron.

Put the disk space check in your webapp logic. If disk space is OK, your user 
has endured one if statement's worth of latency. If disk space is moderately 
low (still have 10 GB), have it run the deletion part in background, and your 
user has endured latency of one if statement and one background spawn. If 
disk space is critically low, throw up a page telling him to wait for disk 
maintenance, and run the deletion in the forground. User endures big latency, 
but at least his work doesn't get garbled. Presumably this won't happen 
often, because it will be caught before it gets critical.

I think I'd build in hysterisis like a thermostat. Run the deletion program at 
7GB free, and have it shut off at 14GB free. That way it won't run too often, 
and won't oscillate as new files get written during its lifetime. Also, as 
the deletion program starts, have it set a flag so that no other deletion 
program starts during its lifetime. Upon the deletion program's termination 
(all termination points), have it unset the flag so other deletion programs 
can be run.

Once again, if it were me, I'd write the program in Perl or Python, whether 
the program is spawned as a cron job or from the web app.

HTH

SteveT

Steve Litt
Founder and acting president: GoLUG
http://www.golug.org