Is it possible to have collision layers, like the way raycast or camera layers work?
Check the docs: Unity - Manual: Layer-based collision detection