My solution to this problem was to use a carefully set-out hierarchy of transforms, each controlling a different part of the camera.
In the middle, there is the ‘target’ transform. It follows the location of the object to be orbited, but does not follow rotation (since it controls the rotation of the camera).
On the outside, where the actual camera is, is the ‘zoom’ transform. This is a child of the ‘target’ transform, and should only move its localPosition (to move towards and away from the centre).
If you like, you can put the camera itself on a third transform (for applying extra effects like screen shaking, etc) but for the purposes of the orbit-cam it is not required.
To rotate the camera, use Transform.rotate (not rotateAround, just rotate) in Space.Local by whatever x and y amounts you need, and the outside camera will move around correctly because of transform parenting!
… I assume you don’t need a code sample here?
DISCLAIMER- this is something I made a while back, and I may have forgotten how parts of it worked. If it doesn’t quite work out for you, tell me, and I’ll try to help fix it.