SampleCmp() always returns zero

[RobertMtx]

I have a very basic question about the use of SampleCmp() and SampleCmpLevelZero(). The shadowmap sample uses these functions, but when I try to use them, I always get a return value of zero. I've tried everything you can imagine, but always get back a zero.

I'm using an immutable sampler in the pixel shader (DX12) to call the method. Generally, I have the sampler set to clamp all axes and with linear filtering and comparison mode set to less than. But even when I set the comparison mode to always, I still get zero.

Is there any reason something like this would be happening that comes to mind? I've verified that the texture and surrounding code is all working correctly by doing a manual comparison (return ShadowPos.z < shadowDepth.Sample( RenderSampler, ShadowPos.xy ).r + 0.0002), and everything works as expected, but any call to SampleCmp returns zero (eg, return SampleCmpLevelZero(ShadowSampler, ShadowPos.xy, ShadowPos.z ).

I've run out of ideas as to what might be causing this. I thought maybe my sampler in code is not linking up correctly with my shader sampler, but that should throw an error. Any ideas what might be going wrong here?

Edit: this is how I've defined these elements in code:

Texture2D shadowDepth;	
SamplerComparisonState ShadowSampler;

Edit 2: I'm getting some really strange behavior from further testing, such as this:
return BlankNormalMap.SampleCmpLevelZero(ShadowSampler, float2(0.0,0.0), -99999999.0 );, which always returns 0.5. Always, regardless of what coordinates I use for the lookup, or what comparison value I send it, or even what comparison mode I use (never, always, less). It just always returns 0.5 (update: similarly, if I try the same code with a pure white texture, I get 1.0 return values). I'm frankly baffled here. Would it be possible to make a mistake that would cause this bahavior? I'm assuming (and hoping) that I'm just overlooking something dumb here, but I can't figure out what it is if that is the case.

[TheMostDiligent]

Does shadow map tutorial work properly?

[RobertMtx]

Yeah, it works great. For whatever reason, in my setup, the SampleCmp() functions appear to be returning a sampled value from the texture instead of actually comparing anything. But almost like a "default sample", or a default color. It returns 0,0 for the shadowmap depth buffer, even if I make sure every pixel in that texture is filled with 1.0. I was thinking maybe the sampler configuration was being miscommunicated, such as setting the filtering mode to some specific thing, but I can think of no configuration that would give me the results I'm seeing (where it returns 1.0 for a white texture, and 0.5 for a normal map, and 0.0 for the depth buffer).

[RobertMtx]

Any ideas as to what would cause such strange behavior? Or any ideas on what I could try to narrow down the cause?

I honestly can't imagine how Diligent Engine would be doing anything that could cause this. From the testing I've done, it appears as if SampleCmp() is completely broken. I guess it's possible the dxc compiler could be to blame. Unfortunately, it would not be an easy thing to compile the shader code in some other way, since my engine saves and loads its own binary data for shaders.

[TheMostDiligent]

I would recommend copying the working version from the shadow map tutorial exactly, and modifying it step-by-step to see at which stage it stops working.

[RobertMtx]

I have checked that code and double checked, As well as compared my version to every version I can find online. They are using the function identically, apart from the fact that some people sample the red channel from the result (SampleCmp().r) where others, including tutorial 13, just use the entire result. But I've tried both of these, and it makes no difference.

I can manually compare the same texture data and everything works perfectly with a normal sampler, and I can plug known-working inputs into SampleCmp() and it does not work as expected. I'm having a hard time imagining how any type of mistake would even be capable of producing the results I'm seeing. What would make SampleCmp() return 1.0 with compare mode set to never? I've tried creating multiple comparison samplers, with every conceivable configuration, but I get the same return values with all of them, which appears to strictly depend on the texture I'm sampling. It doesn't add up.

For the record, I'm laughing on the inside. Chaos makes life interesting, I guess.

[RobertMtx]

Sorry, I didn't mean to sound ungrateful, in case it came across that way. I appreciate any advice you can throw at me. I'm just having a really tough time imagining how any mistake on my part would lead to this function behaving this way. Even if I made every mistake imaginable, and the sampler was set to the wrong mode, SampleCmp() should not be capable of returning a 0.5 for a texture that is filled with a solid color, as is the case with that normal map I tested. That's right, isn't it? If I understand how SampleCmp() works (is supposed to work), it samples 4 points, compares against the input value, then returns a sort of average of those results? So the only way it could possibly return 0.5 is if half of the pixels were calculating a different result than the other half, which should not be possible when all of the pixels are identical.

[TheMostDiligent]

I can't think of anything obvious which is why I am suggesting to copy the working version verbatim and start changing it step-by-step until you find at which step it breaks

[RobertMtx]

Unfortunately, it uses separate pipeline states, and my engine uses a shared global resource signature. That fact makes it difficult to test things. If I remember correctly, the only sample that uses the shared pipeline architecture is the ray tracing sample, but it won't run on my hardware. If you ever find the time, some basic tutorials with shared resource signatures would be really helpful. I assume most complex game engines will need to use them, so that many entities can share resources. Although its been a while since I used anything else, so its hard to remember all of the differences.

But as far as the issue - I'm not going to say I'm sure its not something I've caused, but if it is, it would be something that exclusively makes SampleCmp() or SamplerComparisonState misbehave, because absolutely everything else works correctly, and I've tested every single thing that is interacting with that function and sampler, and everything else is as expected. I mean I can even write the code manually that the function is supposed to do, and it works. The shadows are a little blobby and steppy without linear interpolation, but it works as expected.

I guess another good test would be to use the comparison sampler with another sampling function, to see if it is the sampler or the function that is not working correctly. But I don't think any other sampler functions exist that use it.

[TheMostDiligent]

Unfortunately, it uses separate pipeline states, and my engine uses a shared global resource signature

You can modify the sample to work it like your app - if that does not work, then either there is something wrong in your setup or that might be a bug, but then the issue will be isolated in a small application.

and I've tested every single thing that is interacting with that function and sampler, and everything else is as expected

I've been in this situation a numerous times and I always do exactly like I suggested - start with something that works and move the two implementations towards each other. At some point the issue becomes clear. I really don't know what the problem is. SampleCmp is supposed to work and there is an example where it works so I can only suggest once more to start experimenting with the working version.

[RobertMtx]

I'm still trying to narrow down the problem with this, but I did stumble onto a similar post online where a DX11 user was getting depth value reads from SampleCmp() instead of true/false reads, and it turned out he set up his filter incorrectly. That leads me to believe that any type of issue with the sampler properties may cause SampleCmp to do something weird and unexpected - something its not designed to do. Apparently the function does some type of "I can't do my job, so I'll do this instead". I really need to learn to use the pix debugger, or some other shader debugger. It would probably be very useful in situations like this.

[RobertMtx]

Well, I figured it out. And wow do I feel like an idiot. While looking over the shadowmap tutorial again, for the 20th time, I finally noticed it is using a specific type of linear filter for shadowmap sampling: Diligent::FILTER_TYPE_COMPARISON_LINEAR. Apparently, my brain was ignoring everything except the linear suffix. Unbelievable that I did not notice that. My engine has its own set of defined filter enums, so I just set them to linear to match the sample and moved on. I don't really understand why its possible to set a comparison sampler to a non-comparison filter type without getting some kind of shader error, but I'm just happy to be moving on to new problems.

[TheMostDiligent]

Yes, Diligent does not indeed check the type of the sampler, and only the Direct3D11 validation complains about the mismatch:

D3D11 ERROR: ID3D11DeviceContext::Draw: The Pixel Shader unit expects a Sampler configured for comparison filtering to be set at Slot 0, but the sampler bound at this slot is configured for default filtering.  This mismatch will produce undefined behavior if the sampler is used (e.g. it is not skipped due to shader code branching). [ EXECUTION ERROR #390: DEVICE_DRAW_SAMPLER_MISMATCH]

Both D3D12 and Vulakn validation are OK with the mismatch.
Anyway, glad you figured that out!

[RobertMtx]

Yeah, I wouldn't have expected Diligent to complain. It makes sense to allow combinations of system elements that may not be orthodox at the time, because something that may seem crazy today could become useful tomorrow. But based on the behavior I got from the sampler comparison function, a runtime warning would have been more useful than such weird sample results. Even now, knowing all of the facts, it still doesn't make sense to return 0.0 when sampling a depth map that is full of max depths or 1.0. Maybe the function has some hidden usage when combined with normal sampling filters.