Using shadow texture to recieve shadows (on grass)

Hi everyone!

I’ve created some grass on a custom mesh terrain (NOT Unity terrain) and I want it to receive shadows from surrounding trees. Grass is created from a point cloud where each vertex becomes a billboard in geometry shader. Trees successfully cast shadows on terrain.

I’ve tried sampling Unity’s shadow textures for light (since they have all that fancy good looking cascading stuff) to render shadows on grass but I have failed. The closest question to what I’m asking is this one: Accessing shadow texture in shader - Unity Answers
For some reason neither of the answers given there helped me. I’ve tried both with SHADOW_COORDS and LIGHTING_COORDS (and other appropriate functions) but it didn’t work. I always receive shadow attenuation equal to 1.

I’ve never created a multiple pass shader so if an answer is somewhere in there I am kindly begging for detailed info.

At the bottom is shortened grass billboard shader with a screenshot of a scene.(I deliberately exaggerated with light intensity and shadow strength so that it would be obvious). I’m hoping to achieve different attenuation for my grass in shadow and lit areas. As commented in a shader sample, red color means that attenuation is equal to 1 and every bit of my grass is red. (I reduced the contrast of the picture so it doesn’t hurt your eyes)

How can I use shadow textures to sample light/shadows? Can it be done even though I have a geometry shader? Can I sample shadow texture in vertex shader and pass its value to fragment shader trough geometry shader?

Thank you

Shader and screenshot:

46916-atten-always-one.png

Shader "Custom/GrassBillboardShortened" 
{
	Properties 
	{
		/** properties **/
	}

	SubShader 
	{
		Tags { "Queue" = "Geometry" "RenderType"="Opaque" }

		Pass
		{
			Tags { "LightMode" = "ForwardBase"}

			CGPROGRAM
				#pragma vertex vertexShader
				#pragma fragment fragmentShader
				#pragma geometry geometryShader
				#pragma multi_compile_fwdbase

				#include "UnityCG.cginc"
				#include "AutoLight.cginc"

				struct VS_INPUT
				{
					float4 vertex : POSITION;
					/* .. and other variables used for creating billboard ... */
				};

				// I put shadow stuff in separate struct because of naming in some macros
				struct VS_SHADOW
				{
					float4 pos : POSITION;
					SHADOW_COORDS(0)
				};

				struct GS_INPUT
				{
					float4 worldPosition : POSITION;
					/* .. and other variables like billboard size etc... */
				};

				struct FS_INPUT
				{
					float4 position	: SV_POSITION;
					fixed2 uv_MainTexture : TEXCOORD0;
					/* I used variable name _ShadowCoord since SHADOW_ATTENUATION uses it (according to "AutoLight.cginc") */
					float4 _ShadowCoord : TEXCOORD1;
				};

				/* ... declarations of variables ... */

				// Vertex Shader ------------------------------------------------
				GS_INPUT vertexShader(VS_INPUT v)
				{
					GS_INPUT vOut;
					
					vOut.worldPosition =  mul(_Object2World, v.vertex);

					// I put shadow stuff in separate struct
					VS_SHADOW shadow;
					shadow.pos = mul(UNITY_MATRIX_MVP, v.vertex);

					/* I used this instead of TRANSFER_SHADOW(shadow) since I had to store shadow coords in a variable to pass them to
					 * geometry shader. When I use TRANSFER_SHADOW, I guess shadows coords are stored somewhere in _ShadowCoord variable
					 * according to "AutoLight.cginc" but for some reason I can't reference it with shadow._ShadowCoord */
					/* It's not that I wanted to have it this way but every other attempt (like sampling attenuation here in vertex shader)
					 * also failed */
					vOut.shadowCoords = ComputeScreenPos(shadow.pos); 

					return vOut;
				}


				// Geometry Shader -----------------------------------------------------
				[maxvertexcount(4)]
				void geometryShader(point GS_INPUT p[1], inout TriangleStream<FS_INPUT> triStream)
				{
					/* ... bunch of code for creating billboard geometry - that works ... */
					float4 v[4];	// 4 vertices for billboard are created 
					float4x4 vpMatrix = mul(UNITY_MATRIX_MVP, _World2Object);
					
					FS_INPUT fIn;
					fIn.position = mul(vpMatrix, v[0]);
					fIn.uv_MainTexture = float2(1.0f, 0.0f);
					fIn._ShadowCoord = p[0].shadowCoords;
					
					triStream.Append(fIn);

					fIn.position =  mul(vpMatrix, v[1]);
					fIn.uv_MainTexture = float2(1.0f, 1.0f);
					fIn._ShadowCoord = p[0].shadowCoords;

					triStream.Append(fIn);

					fIn.position =  mul(vpMatrix, v[2]);
					fIn.uv_MainTexture = float2(0.0f, 0.0f);
					fIn._ShadowCoord = p[0].shadowCoords;

					triStream.Append(fIn);

					fIn.position =  mul(vpMatrix, v[3]);
					fIn.uv_MainTexture = float2(0.0f, 1.0f);
					fIn._ShadowCoord = p[0].shadowCoords;

					triStream.Append(fIn);
				}


				// Fragment Shader -----------------------------------------------
				float4 fragmentShader(FS_INPUT fIn) : COLOR
				{
					fixed4 color = tex2D(_MainTex, fIn.uv_MainTexture);
					
					if (color.a < _Cutoff) 
						discard;

					float atten = SHADOW_ATTENUATION(fIn);
					if (atten > 0.99)
						color = fixed4(1.0,0.0f,0.0f,1.0f);
					else
						color = fixed4(0.0,1.0,0.0,1.0f);

					// I always get fully red color which means atten is always > 0.99
					return color;
				}

			ENDCG
		}
	} 

	FallBack "Diffuse"
}

Yes, it can be done with geometry shader!

I’ve managed to solve this problem with the help of this post:
http://forum.unity3d.com/threads/is-it-possible-to-use-geometry-shader-with-surface-shaders.164905/#post-1658095

There are couple of things you need to do to make shadows right and it’s quite easy once you find out what Unity expects you to do.


BASE PASS

  1. At the end of your vertex shader you need to call TRANSFER_VERTEX_TO_FRAGMENT(f), where f is your fragment shader input structure. If you’re creating geometry in geometry shader, then you need to do it in geometry shader for every vertex created there. You also need to include “AutoLight.cginc” which contains that macro.
    NOTE: That macro will use f.pos so you have to provide SV_POSITION variable named pos. Example:

           struct FS_INPUT  // fragment shader input structure
        {
           	float4 pos : SV_POSITION;  // has to be called this way because of 
                                          // TRANSFER_VERTEX_TO_FRAGMENT macro
           	LIGHTING_COORDS(0,1)       // data needed to sample light is declared on 
                                          // TEXCOORDS 0 and 1
        };
            
        FS_INPUT vertexShader(VS_INPUT vIn)
        {
            FS_INPUT f;
            f.pos = mul(UNITY_MATRIX_MVP, vIn.position); // classic model-view-projection
            TRANSFER_VERTEX_TO_FRAGMENT(f);      // initialization of data needed for sampling light
            
            return f;
        };
    
  2. In the fragment shader you’ll sample light with:

    float atten = LIGHT_ATTENUATION(fIn); // fIn is input fragment.
    
  3. I did my own lighting calculation based on (ambient + atten * lambert) * _LightColor0.rgb, but I guess it depends on what you want.

  4. If you are not creating geometry in geometry shader, then adding Fallback "Diffuse" should add proper shadows to your model (because it has ShadowCaster and ShadowCollector passes).


SHADOWCASTER PASS

If you’re creating your geometry in geometry shader, then you need to create your own ShadowCaster pass. For some reason ShadowCollector pass is not needed? It works for me perfectly without shadow collector. In “UnityCG.cginc” there are “deprecated shadow collector pass helpers”. Is ShadowCollector deprecated and not needed anymore?

Things you need to do:

  1. Add tag: Tags { "LightMode" = "ShadowCaster" }

  2. Instead of TRANSFER_VERTEX_TO_FRAGMENT(f) use TRANSFER_SHADOW_CASTER(f). Example:

       struct SHADOW_VERTEX // helper struct because TRANSFER_SHADOW_CASTER uses v.vertex
    {
    	float4 vertex : POSITION; // local position of vertex
    };
    struct FS_INPUT
    {
    	V2F_SHADOW_CASTER;
    };
    
    //somewhere in geometry shader:
    
    FS_INPUT f;
    SHADOW_VERTEX v;
    v.vertex = // local position of new vertex;
    TRANFER_SHADOW_CASTER(f)		// uses "v.vertex"
    
  3. In fragment shader use SHADOW_CASTER_FRAGMENT(f):

       fixed4 fragmentShader (FS_INPUT fIn) : COLOR
    {
        SHADOW_CASTER_FRAGMENT(fIn)
    }
    

And that should be it! I hope this will be helpful to someone. Here’s a screenshot of this working:


SOURCE:

Here’s my shader’s source. You can use it wherever you want. You don’t have to give me any credits, however they will be warmly welcomed. Enjoy!
[54417-grassbillboard.txt|54417]