How to cull/render to through a 'window'?

I need to render a bunch of objects but ‘mask’ them off if they are behind a ‘window’. That is, I need a rectangular ‘hole’ through which objects behind it can be seen as if through a window, objects in front are rendered normally. Like if I had a ‘wall’ but the wall itself is transparent. I hope that made sense?
19019-culling.png
Note that the ‘window’ may be a quad, or a hole in a wall (plane), which ever will work best.

What you want to do is no simple task. You want a dimensional portal effect. I really think you’ll want to use stencil buffers. They are made just for this sort of thing.

Think of a stencil buffer as very similar to a depth buffer but used to stencil objects in the scene.

When you render to a depth buffer, it takes all of the objects in front of your camera (visible, on same layer, etc, does some matrix math in the pipeline), and puts the color from each object along your view into the depth buffer (i.e. the image you see on screen).

Now consider a stencil buffer. This does the same thing, but like a depth buffer you control when the buffer gets written to and what you render into it. And it is easier to set the value that gets stored in it (whereas the depth buffer will get the color by default unless you write your own pixel shader).

Now for the cool part, so you set up a stencil buffer, turn it on, set it to set each pixel location in the buffer that gets drawn to a value of 1, draw your square window in the center of the image above (the window where things will be visible) as a solid opaque quad, then turn the stencil buffer off. Now your stencil buffer contains a stencil of your window with values set to 1.

Now when you render the hidden objects, you can tell them through the shader to only render where the stencil buffer is equal to 1.

Now I know how I would do this in the old school fixed pipeline. You would turn off the depth buffer, turn on the stencil buffer, call your Draw() call for the Window quad (or any other window objects to render to stencil if you’d like), turn off the stencil, then turn on a state in the pipeline saying “Only draw where stencil equals 1”, then call Draw() on all of the “hidden” objects. Next, disable the stencil equal 1 test, and draw everything else.

You basically would do the same thing in Unity with the shaders.

Create a shader for your Window object that disables the depth buffer, enables the stencil buffer, and sets the stencil value to 1. Create a shader for the hidden objects where the stencil buffer TEST is enabled, with a test of “equal 1”. All other shaders remain normal.

Lastly, you would need to set the RenderQueue order so that the window object always draws first. No need for extra textures, no need for extra cameras, all done in one pass. The only downside is the large stencil buffer, but the game engine may create this even if you don’t use it.

Here’s a link to shader lab:

Here’s a link for doing stencil in SL:

If you want to only see objects in front of the quad object, be sure your Window shader is set to Cull back side, be sure your quad is front facing, and don’t use a cube or other closed object. Now you will have a 1 way portal.

I didn’t make this an answer since you will have to research yourself, but hopefully it will guide you.

I promised to create a quick tutorial on this and time went by, so I apologize. I just ran through it tonight and it was easier than I thought.

I created two materials, Hidden (which is applied to the objects to hide) and Portal (which is applied to a quad that will be your “window”.

I created two shaders of the standard affair and modified as follows:
Hidden Shader:
Just under the auto generated shader line “LOD 200”, add:

Stencil {
Ref 1
Comp Equal
}

Portal Shader:
Just under the auto generated “LOD 200”, add:

ZWrite Off
ColorMask 0
	
Pass {
	Stencil {
		Ref 1
		Comp always
		Pass replace
	}
}

This is essentially the same concept I discussed above. The Portal shader turns off writing to the depth and color buffers (so it’s invisible) and turn on writing a value of 1 to the stencil buffer. The “Comp always” means the stencil test always passes. The “Pass replace” means to replace the value in the stencil buffer with “Ref” when the test passes (which is always). So now the stencil buffer is 1 where your portal window is.

The Hidden shader just says to only pass the pixel color on through the pipeline if the stencil buffer value Equal 1.

You can now slide the portal around or move the camera, and only see the portion of the Hidden objects behind the quad. If you assign the Portal shader to a Quad, it’s a “portal” effect. If you assign it to a cube, you would only see anything behind the cube (Note: If you wanted to only see parts INSIDE the cube, it would require some extra playing with the stencil buffer like adding/subtracting and using the depth tests with the stencil). Same goes for basically any mesh you assign it to, only shows the parts of the Hidden objects that lie behind any pixels where the mesh would normally be seen is the general rule of thumb.

Please note that this assumes shaders all start out with the default settings. If you want to add this capability to an existing shader, you would need to be careful about leaving stencils floating around, etc. You may also need to clear your stencil buffer after using this if anything else uses it, which many times shadows do.

I also did not set the queue order and am probably getting lucky that the Window shader is getting drawn and processed BEFORE the hidden objects, probably because I created the window quad first. You would want to adjust the queue to make the window draw first to update the stencil buffer first, or you would not see anything at all.

Here’s a quick snapshot from the Editor camera:

Here’s a thought: what if you made your wall pure green. Render as usual, but to a RenderTexture. Then run a green-screen shader on that to turn the green into transparent.

Hi everyone!I have a problem with these shaders I can’t solve.

I am making a game with mirror fragments. Since render texture is a bit too heavy (I have like 10 fragments and maybe want to add more), I duplicated and mirrored the scene in 3DS max.
I put these shaders (hidden & portal) on the fragments and on the objects behind it. This works, but when these fragments move, the portals allows for the camera to see the other objects also from other fragments. They ‘override’ if I explain that properly:


The problem is, how can I create multiple portals for different objects? I have never written shaders before so please help me if I need to add something there.

Thank you!