How do I make touch ignore specific layers?

In my game I have tiles the player needs to click and drag around. However, sometimes there are things on top of the tiles (creatures, gems etc) and their collision volumes block player input.

I cannot put all these other things in the ‘ignore raycast’ layer, because they need to be in a variety of different layers in order to correctly manage collisions between them. I just want to be able to remove all but one or two layers from interacting with touch, and it doesn’t seem to be possible. I can’t believe there isn’t a LayerMask somewhere for this purpose!

Thanks in advance.

As it turns out there IS a layermask for exactly this purpose, only it’s not exposed in the editor and has to be set from script: eventMask on the camera that’s displaying the scene.