Raycasting question

I have a character that I want to track his eyes to where the mouse is pointing. To do this I set up a box collider and positioned it in the camera's view (see screenshot). I wrote a script for the collider to raycast from the mouse and point the eyes at the result. This works great.

However, the collider is preventing anything else from being raycast onto because it's blocking the whole scene. I have the character set up to do an animation if clicked, but it doesn't work while I have the collider box there.

Is there a way for me to only enable the collider box right before I raycast, then turn it off? I have tried setting hideFlags, but that didn't work. I also tried moving the collider box out of view, then temporarily back into view for the raycast, but that didn't work either.

Any ideas? Seems like this should be pretty straightforward.

Raycasting

Do you need to use a collider? I would suggest converting the mouse position to a world position, then have the eyes LookAt that point. If you need the collider, maybe putting it in a different layer would help.

There is a layer called ignoreRaycasts or something you could use. At the start of a frame, set the collider's layer to a raycast-able layer, track the eyes to the mouse, then set the collider layer back to ignoreRaycasts.