There's an OnMouseExit event which you can use just like OnMouseUp, to trigger code when the mouse is no longer over the object. You'd probably have to use this in conjunction with the mouse up/down tests to get the effect you want.

(note, the comments appear to be wrong in the example on the documentation page linked above)

Here's an example which activates an object when clicked, and deactivates when the mouse is released, or when the mouse exits. Notice that it also reactivates if the mouse is brought back over the object while still held down (standard behaviour for normal Mac and Windows UI buttons). I'm using a boolean variable called 'wasClicked' to remember the state, to achieve this. The Activate() and Deactivate() functions are custom functions - I have just added code which makes the object change colour as an example, but you can put whatever you want in there.

var wasClicked : boolean;

function OnMouseDown() {
    wasClicked = true;
    Activate();
}

function OnMouseUp() {
    wasClicked = false;
    Deactivate();
}

function OnMouseEnter() {
    if (wasClicked) {
        Activate();
    }
}

function OnMouseExit() {
    Deactivate();
}

function Activate() {
    renderer.material.color = Color.red;
}

function Deactivate() {
    renderer.material.color = Color.white;
}

Thank you so much! Very useful.