Moving the camera is an unreliable solution, since it accounts poorly for geometry that might be in the way; the camera migh get moved beyond hilltops or the walls of houses and make you see behind things when zoomed which are in front of the camera while unzoomed.

Changing the FOV is the only reliable way to implement zoom, and thankfully, also happens to be what a real camera does when it zooms. :) A physical camera changes the focal length of its lens, which in turn changes the angle of view, which corresponds to the FOV of a computer graphics camera.

Also, increasing the field of view does not zoom in, it is the other way around. :) "Zooming in" occurs when the FOV is changed to something smaller than the camera's default "zoomed out" FOV. This makes intuitive sense, because the size of an object will take up more of the viewport's area, if the angle of view is smaller. Conversely, if the angle of view is greater, then the same camera will fit more of the world into its viewport, and thus, the individual object in the world must become smaller.

The distortion you describe sounds like what happens when the FOV approaches 180 - the perspective curves and becomes twisted? I'm not 100% certain what you mean by stretching and distorting when zoomed in, but perhaps this could be caused because you're zooming very close to 1 when you zoom in? Try to play around with the FOV with a camera in the editor and study the camera's frustum as you change the FOV. Notice that the view frustum becomes very long and thin when the FOV is low? This happens because the far plane is at the same distance from the camera, so to reach it with such a low FOV, the frustum must become a very thin, long pyramid. This means that vectors along one side of the frustum to the camera become almost parallel to vectors from the middle of the frustum to the camera, i.e., the camera basically approximates orthogonality at extremely low FOVs. Depending on how geometry is positioned relative to the camera, this might stretch the objects in the sides of the screen, because the camera's projection loses its perspective correctness.

Keep with the FOV solution for zooming, but perform the zooming within a range of values that are still comfortably far from 1 and 180. Something like zoomed out = 90 and zoomed in = 40, or maybe 60/30.