Thanks for your help, Brian, you got me on the right track.

I ended up creating a script that I think could be helpful for everyone who asked a similar question as I did on the forum or here.

Just add this to the object you want to rotate and tell it what camera you are using and the distance you want the rays to go.

I am actually a cleaner version of this that slerps the rotation and actually just projects the mouse touch in front of the object so it is smoother as your mouse goes off of it but this was the easiest way to show the example.

It boils down to this:

1. Add a looker object and set it's parent to be "this"
2. On Mouse down, shoot a ray then have the Looker Object "Look at" that point
3. On Mouse Move, Have the Looker Object look at the new point.
4. Rotate the object, relative to the world coordinates, the difference of the Euler rotations of the starting and currently Lookers.

For me, the key was Euler math couldn't be easier. Then, once the difference is calculated (with simple subtraction) rotate the object just that little bit.

It works! Please let me know your experiences!

Best,

Thom

using UnityEngine;

using System.Collections;

public class SimpleRotate : MonoBehaviour {

public Camera MyCamera;
public float Distance = 100f;
Vector3 LookerVector;
Vector3 LookerVectorStart;
bool FreeRotating;
GameObject Looker;
void Start(){
	Looker = new GameObject();
	Looker.transform.parent = transform;
}
void Update () {
	Ray Mray = MyCamera.ScreenPointToRay(Input.mousePosition); 
    RaycastHit Mhit = new RaycastHit(); 

	if(Input.GetMouseButton(0))
	{
   		if (Physics.Raycast(Mray, out Mhit, Distance)) 
		{
			GameObject hitObject = Mhit.collider.gameObject;
			if( hitObject == gameObject )
			{
				if(Input.GetMouseButtonDown(0))
   					StartFreeRotate( Mhit.point);					
   				else
           			FreeRotate(Mhit.point);
			}
			else
				FreeRotating=false;		
		}
	}
	if (Input.GetMouseButtonUp(0))
    	FreeRotating=false;			

}
public  void StartFreeRotate(Vector3 CurrentPos)
{
	Looker.transform.LookAt(CurrentPos);
	LookerVectorStart = Looker.transform.rotation.eulerAngles;
	FreeRotating=true;	
}
public  void FreeRotate(Vector3 CurrentPos)
{
	Looker.transform.LookAt(CurrentPos);
	LookerVector = Looker.transform.rotation.eulerAngles;
	Vector3 LookOffset = (LookerVector -LookerVectorStart);
	// Not sure why x needs to be negative but it does for me!
	LookOffset = new Vector3 ( -LookOffset.x, LookOffset.y, LookOffset.z);
	transform.Rotate( LookOffset, Space.World);
	LookerVectorStart = LookerVector;

}

}

For what you describe, you could use:

Vector3.Angle(mousePosition-objectPosition,newMousePosition-objectPosition);

You may have to play with the mouse position a little bit to figure out how much you really wanted to move the object. I'm not sure which plane you should be operating on.

EDIT: I think you probably want to raycast at the object, figure out what point you clicked on, and use that as your point to follow the mouse. This is going to be a bit tricky - I'll try to get some code working down below.

For momentum, you just need speed of mouse movement at release, here is a basic way to get that, you probably want something smoother.

Vector3 lastMousePosition;
Vector3 deltaMousePosition;

void Update(){
    deltaMousePosition = (Input.mousePosition-lastMousePosition)/Time.deltaTime;
    lastMousePosition = Input.mousePosition;
}

Here is how to get the point on the object you clicked on:

Vector3 down;

void Update () {
    if(Input.GetMouseButtonDown(0)){
        Ray ray = Camera.main.ScreenPointToRay (Input.mousePosition);
        RaycastHit hit;
        if (Physics.Raycast (ray,out hit, 50)) {
            Debug.Log(hit.point);
        }
    }
}

Here is a somewhat fidgety implementation of what you want:

Vector3 drag;
public Transform target;

void Update () {

    if(Input.GetMouseButton(0)){
        Ray ray = Camera.main.ScreenPointToRay (Input.mousePosition);
        RaycastHit hit;
        if (Physics.Raycast (ray,out hit, 50)) {
            drag=(hit.point);
        }
    }

    target.LookAt(drag);
}

To avoid the glitchiness, you can either use a different solution, perhaps faking it, or figuring out mouse position on a plane at the original distance. Or, you could compare RaycastHit.triangleIndex to see if you are still mousing over the same triangle.