The physics joints aren't rigid - they always allow a little spurious movement. Some things you can do to improve this behaviour:

1- Freeze rotations in the other axis, and freeze position in the hinge axis; if your hinge axis is Y, for instance, in the Inspector (Rigidbody/Constraints) freeze rotations in the X and Z axis, and freeze position in the Y axis;
2- If the center object doesn't move, child the rotating object to it, adjust the hinge Anchor to the static object center and let the Connected Rigidbody blank (it will connect to the world) - you can still use the above freezing to get best results;
3- Forget about joints and physics and do it by software - way more complicated, but sometimes is the better solution. It's easier to fake the results than to accurately simulate them.