Handling collision with fast Lerp

I have a game, where the player is an object that is Lerped to the position underneath the mouse. You must dodge oncoming objects.

Problem is, when you move your mouse outside the screen in the top, and then back inside in the bottom, the player will almost instantly Lerp to the bottom, completely ignoring collisions with any objects inbetween the top point where you exited, and the bottom point where you entered.

Do I have any options here, or do I have to hard-code/hack some stupid raycast check before Lerping?

EDIT: I should probably mention, that collision is handled with a rigidbody on my object, as well as on all obstacles. Obstacles are triggers, player is not.

Actually, this fixed the problem:

void MoveTowards (Vector3 pos) {
		Vector3 newPos = new Vector3(pos.x, transform.position.y, pos.z);
		rigidbody.position = Vector3.Lerp(transform.position, newPos, moveSpeed * Time.deltaTime);
	}

Calling the function on the position of the rigidbody instead of the transform.

You’re a champ, thanks mate.