[TriLUG] perl: GET'ing large file with LWP

[Joseph Tate]

I'd use two requests; HEAD for the headers, then GET to get it
afterwards after processing the headers.  Yes, it's a bit redundant,
but otherwise you're writing the HTTP protocol yourself.

On Sun, Sep 20, 2009 at 1:45 PM, Joseph Mack NA3T <jmack at wm7d.net> wrote:
> I can GET an xml file with
>
> $ua = LWP::UserAgent->new;
> $url = 'http::/somewhere/';
> $response = $ua->get($url,
>                        ':content_file' =>
>                        'database.xml'
>                        );
>
> The file takes about 10mins to download and I would like the user to know
> why their application appears to be doing nothing.
>
> Before downloading the file, I'd like to be able to
>
> o retrieve content-type. The content will not always be the file (it could
> be a short html error page saying that your cookie has expired)
>
> o retrieve length ($response->content). This will allow me to put up a
> progress bar.
>
> The only instructions on LWP I've found download the file first and then you
> look at the headers rather than the other way around.
>
> The download is html encoded eg "e; rather than ". I notice that
> lwp-request has an -o format option which I assume can change the output. Is
> there any way of removing the html encoding when downloading with LWP?
>
> Thanks
> Joe
> --
> Joseph Mack NA3T EME(B,D), FM05lw North Carolina
> jmack (at) wm7d (dot) net - azimuthal equidistant map
> generator at http://www.wm7d.net/azproj.shtml
> Homepage http://www.austintek.com/ It's GNU/Linux!
> --
> TriLUG mailing list        : http://www.trilug.org/mailman/listinfo/trilug
> TriLUG FAQ  : http://www.trilug.org/wiki/Frequently_Asked_Questions
>



-- 
Joseph Tate
Personal e-mail: jtate AT dragonstrider DOT com
Web: http://www.dragonstrider.com