The matrix only affects the position and rotation of the handle, not the size. Normally you can specify the size that you calculated with HandleUtility.GetHandleSize but the PositionHandle does that internally.

Here's the (internal) implementation of PositionHandle ( thanks ILSpy ;) )

public static Vector3 PositionHandle(Vector3 position, Quaternion rotation)
{
    float handleSize = HandleUtility.GetHandleSize(position);
    Color color = Handles.color;
    Handles.color = Handles.xAxisColor;
    position = Handles.Slider(position, rotation * Vector3.right, handleSize, new Handles.DrawCapFunction(Handles.ArrowCap), SnapSettings.move.x);
    Handles.color = Handles.yAxisColor;
    position = Handles.Slider(position, rotation * Vector3.up, handleSize, new Handles.DrawCapFunction(Handles.ArrowCap), SnapSettings.move.y);
    Handles.color = Handles.zAxisColor;
    position = Handles.Slider(position, rotation * Vector3.forward, handleSize, new Handles.DrawCapFunction(Handles.ArrowCap), SnapSettings.move.z);
    Handles.color = Handles.centerColor;
    position = Handles.FreeMoveHandle(position, rotation, handleSize * 0.15f, SnapSettings.move, new Handles.DrawCapFunction(Handles.RectangleCap));
    Handles.color = color;
    return position;
}

In my experience changing the matrix doesn't have any effect for many functions (inluding RotationHandle()) and it also bugged some other functions for me, so I stopped using it. Now I draw my own handles and transform everything manually before passing it into Handles' functions.

But it might be quite difficult to do the full rotation handle. In my case I was happy with a simple one using Handles.Disc() together with Handles.DrawLine(), so it might work for you as well.