Jonas - Haskell HTTP Library Improvement

Haskell HTTP Library Improvement

Info

This is an improved version of the current Haskell HTTP Library. The issues of the old library are nasty error handling and the usage of regular Strings ([Char], meaning linked lists of single characters) for representing the body of a HTTP response. This version presents an alternative way of handling errors, using monads and try-catch behaviour, and the payload of HTTP requests and bodies are handled using ByteStrings.

Information about the error handling can be found here.

Problems

One obstacle that is currently unsolved is the way closing of sockets is handled when using lazy byte strings combined with automatically closed HTTP sessions. To put it simple, sockets aren't closed at all because it's unclear when to do so. This is not a problem when using strict byte strings because one can close the socket when the body is downloaded.

Benchmarks

When trying to download a 175MB binary file using the current Haskell HTTP library the test program didn't even start to write to the file before the memory was completely filled and the operating system started to swap like crazy. So instead we compared against downloading using the common utility "wget".

Our test programs uses the lazy version of byte strings. The difference with using the strict version is that the whole file is loaded to the memory and then written (very large memory usage for very large files).

Program	Memory usage	CPU usage	Time
wget	~500kB	~25%	~10s
our test program	~3MB	~15%	~11s

Usage comparison

For an example using the current version see this example file.

For a program doing pretty much the same thing using this improved version:

module Main where

import qualified Data.ByteString.Lazy.Char8 as BS
import Network.HTTP (rspBody)
import Network.HTTP.UserAgent as UA

main = do { rsp <- UA.get "http://www.testurl.com/"
          ; BS.putStr $ rspBody rsp
	  }
       `catch`
       \_ -> putStrLn "Request failed."

Authors

Tomas Schilling <nominolo@googlemail.com>
Jonas Ådahl <tox@dtek.chalmers.se>

Source code

The source code is available for download using darcs. The version using lazy byte strings is available using:

darcs get http://www.dtek.chalmers.se/~tox/darcs/http

and the strict version:

darcs get http://www.dtek.chalmers.se/~tox/darcs/http-strict

Jonas Ådahl