Get a raycast to hit only child objects to use axes

I have walls, floors, and ceilings of a level grouped under the name of Map5 which I have tagged Level. I have also made a rotation script that with the intention of rotating the whole level around the player. I want to use the walls axes for my rotation, which should have a sort of hinge effect.I have tagged the walls based on the direction they are facing (for example; left walls are tagged as Left) The problem I have is that the any wall the ray hits is counted as Level since they are all children of Map5. I need my ray to hit the children so I can use their axes but I don't know how. Please help?

Here is my current code If it helps at all...

var Player : Transform;
var rotateAmount = 90;
var controller : CharacterController;
var Hit : RaycastHit;

function Start() {
}

function FixedUpdate () {   
  //casts a ray from camera to point in world
  // If the controller is grounded and the ray hits an object; then you can Rotate walls
  if (controller.isGrounded)
    canRotate =true;    
if(Input.GetMouseButtonDown(0)){ 
    var ray = Camera.main.ScreenPointToRay (Input.mousePosition);
        if  (Physics.Raycast (ray, Hit, 50)){
        var Hitobject : GameObject = Hit.transform.gameObject;
        var pivot : Transform = Hitobject.transform;
        Debug.Log("hit"+ gameObject.name);
            if((Input.GetMouseButtonDown(0)) && (Hitobject.gameObject.tag == "Left")){
                RotateObject(Player.position,pivot.forward, 90,1);
            }
        }
    }
}

If you are getting the entire level back then you have a collider attached to your entire level. You need to have colliders on your children in order to hit them.

by the way, if you rotate your entire world you lose out on static batching. something to consider.