[TriLUG] wget puzzle

[crimsun at fungus.sh.nu]

On Fri, May 02, 2003 at 02:23:54PM -0400, Andrew Perrin wrote:
> wget -r www.ussg.iu.edu/hypermail/linux/kernel
> 
> but for some reason it downloads only the directories containing messages
> from 2002 on (0201.*-->0305.*).  The directories are present and readable
> on the original site. Any ideas why this might happen?

There's a robots.txt with this section:

User-agent: *
Disallow: /hypermail/linux/kernel/95
Disallow: /hypermail/linux/kernel/96
Disallow: /hypermail/linux/kernel/97
Disallow: /hypermail/linux/kernel/98
Disallow: /hypermail/linux/kernel/99
Disallow: /hypermail/linux/kernel/00
Disallow: /hypermail/linux/kernel/01

Hope that helps.

-Dan

-- 
Dan Chen                crimsun at fungus.sh.nu
GPG key: www.unc.edu/~crimsun/pubkey.gpg.asc