I have a big problem its driving me crazy so im hoping someone will help me :D

I have this game when i press mouse button 1 it instantiates a cube at the position when the ray hits. This part of the program works fine and looks like that :

if(Input.GetButtonDown("Fire1")&& Physics.Raycast (cam.position, cam.forward, hit, 100)!=false )

{

Instantiate(box,point,Quaternion.identity);

}

Then when i press mouse button 2 the object that the ray hits is destroyed. That part works allso well, too well and it looks like this:

if(Input.GetButtonDown("Fire2"))

{
    if (Physics.Raycast (cam.position, cam.forward, hit, 100) )
    {   


        if(hit.transform.gameObject)
        {

        Destroy(hit.transform.gameObject);

        }
    }
}

The problem is when i press mouse button 2 it destroys every object the raycast hits like for example the terrain and i dont want that i want it to destroy only the box and not eany other object and i allso dont want to turn on the Ignore Raycast on eny of my objects.