Try this. it's untested (I don't have Unity in front of me right now)

Ok, now implemented and tested, and turns out it was a little more complex to implement - mainly because at a high framerate, you get many frames where the mouse axis input is zero during a drag - so if you don't keep track of speed over a number of frames and you release the button during one of these 'zero value' frames, it appears as though the mouse wasn't moving!

First, the rotation is done in Update rather than MouseDrag, so that it can continue after the mouse is released. I'm using OnMouseDown to detect that this object was originally clicked on, and subsequently using Input.GetMouseButton to determine that the mouse is still held down.

Basically, while dragging, it keeps a separate "average speed" value (avgSpeed), which contains the current speed of rotation averaged over the last half-second or so (Using Lerp). It then uses this value when the mouse is released as the speed to continue rotating, and gradually Lerps this value down to zero to slow the rotation.

I have also used a different method of rotating the objects so that the rotation direction is always in accordance with the direction that you're dragging the mouse in screen-space.

Hope this fits what you're looking for!

var rotationSpeed = 10.0;
var lerpSpeed = 1.0;

private var speed = new Vector3();
private var avgSpeed = new Vector3();
private var dragging = false;
private var targetSpeedX = new Vector3();

function OnMouseDown() 
{
    dragging = true;
}

function Update () 
{

    if (Input.GetMouseButton(0) && dragging) {
        speed = new Vector3(-Input.GetAxis ("Mouse X"), Input.GetAxis("Mouse Y"), 0);
        avgSpeed = Vector3.Lerp(avgSpeed,speed,Time.deltaTime * 5);
    } else {
        if (dragging) {
            speed = avgSpeed;
            dragging = false;
        }
        var i = Time.deltaTime * lerpSpeed;
        speed = Vector3.Lerp( speed, Vector3.zero, i);   
    }

    transform.Rotate( Camera.main.transform.up * speed.x * rotationSpeed, Space.World );
    transform.Rotate( Camera.main.transform.right * speed.y * rotationSpeed, Space.World );

}

This is exactly what I was after, however, is it much hassle to change it so that instead of click-drag I can use the arrow keys?