How Do I make it harder to move rigid body?

In my game I Have a castle made out of rigid body blocks.
I don’t want to knock down the blocks using my musket but I want to with my cannon

So how do I Make it so rigid bodys wont move as much?

Javascript only please.

Trying adding mass to the Rigidbody and then lower the power of your musket and then the increase the power of your cannon ball.
And for physics you can use metal for the Rigidbody blocks.