z and w output of UnityObjectToClipPos()

Hi.
As we are instructed to migrate to UnityObjectToClipPos() instead of doing the matrix multiplication ourselves, we are changing our shader and some of our previous calculation relies on the z and w output of the MVP transform was broken.

As far as the document says, Unity uses the openGL style projection matrix, which looks like:

[ 2n/r-l     0      r+l/r-l        0
    0      2n/t-b    t+b/t-b        0
    0         0      -(f+n)/f-n  -2fn/f-n
    0         0        -1           0    ]

Let’s assume we have an input vector in.vertex, and z on the RHS below corresponds to in.vertex’s z coordinate in the camera space. If the above matrix is to be taken in its literal sense, after we do

out.vertex = UnityObjectToClipPos(in.vertex);

The content of out.vertex should be

out.vertex.z = -(z*(f+n) + 2fn)/(f-n);
out.vertex.w = -z;

This doesn’t seems to be the case. After some hacking it seems we’re getting:

 out.vertex.z >> 1 (z >> near plane), >>0 (z >> far plane);
 out.vertex.w = +z;

in the output of UnityObjectToClipPos(). Note that if the openGL textbook math is actually used, out.vertex.z’s absolute value should increase while z’s absolute value is increasing, regardless of z’s sign on the RHS and the output’s sign.

We’re wondering if anyone from Unity can confirm this? Our calculation can be easily fixed as long as we know the exact formulae under the hood. Thanks.

Well, what you have to understand is that camera space is a right-hand space while object space in Unity is a left-hand space. In camera space the z value is always negative. That’s why you have a “-1” in the third column fourth row instead of a “1” to get the new “w” value into the positive.

Keep in mind that the GPU does perform the perspective / homogeneous divide after your shader. It will bring the “w” component back to “1” and the “z” that is written to the depth buffer is:

-(z*(f+n) + 2fn)/((f-n)*w)

or

(f+n)/(f-n) - 2fn/((f-n)*w)

(If i calculated correctly) which basically results in a value between “-1”(near distance) and “1”(far distance).

Example:

// at near distance you get:
(f+n)/(f-n)-2fn/((f-n)*n)  --> (f+n)/(f-n)-2f/(f-n)  --> (f+n-2f)/(f-n)  --> (n-f)/(f-n) --> -1
// at far distance you get:
(f+n)/(f-n)-2fn/((f-n)*f)  --> (f+n)/(f-n)-2n/(f-n)  --> (f+n-2n)/(f-n)  --> (f-n)/(f-n) --> 1

I recently posted a more detailed answer on how the projection matrix works. I’ve linked this paper from my other answer