OnMouseExit fires every frame when the mouse is down?

I have a model with a rigidbody attached and I change the material in OnMouseEnter and OnMouseExit. It works correctly when I don't hold the mouse button down, but when I do have any mouse button held down and I drag over the object it flickers. Is this a known bug or am I doing something wrong?

Edit: added code

void OnMouseEnter()
{
    if (sCurrentHoverSlot == null)
    {
        sCurrentHoverSlot = this;
        gameObject.renderer.material = GUIManager.instance().weaponSlotHoverMaterial;
    }
}

void OnMouseExit()
{
    if (sCurrentHoverSlot == this)
    {
        sCurrentHoverSlot = null;
        gameObject.renderer.material = GUIManager.instance().weaponSlotDefaultMaterial;
    }
}

It's a bug that happens if you're using OnGUI.

As a side note, you don't need a rigidbody if you're just using OnMouseEnter/Exit, all you need is a collider. If you are using it as a moving object under physics control, never mind.

Do you have something inside any of the OnMouseDown/Over/Drag functions?