Not satisfied with stencil shadows

BlitzMax Forums/OpenGL Module/Not satisfied with stencil shadows

JoshK(Posted 2006) [#1]
I implemented stencil shadows, but I am not really happy with them. Shadow go through solid objects, and appear on surfaces that face away from the light source. They just don't seem like a good technique for real-time programs.

I am using lightmaps for brush solids, hardware lights to illuminate moving objects, and vertex colors for static meshes. Stencil shadows are used on movable physical objects. There are so many errors that can occur if you cast shadows on walls, that it makes everything look really unrealistic.

What can I do to improve these for real-time use? I thought about just casting shadow downwards away from the light source, just to have simple ground shadows.






sswift(Posted 2006) [#2]
The problem is you're taking a half-assed approach to stencil shadows.

There are two halves to your problem here.

The first half is that you are not rendering the shadows correctly. Imagine rendering all the shadows first, pure black, onto a white background. If they overlap, it doesn't matter. That area is pure black no matter what. After you're done rendering the shadows, then you can brighten them up a bit. Maybe make them fall off from the light source by distance. Don't ask me how to do this, I don't know.

The second half of your problem is that in Doom, EVERYTHING cast shadows from EVERY light source. Assuming you have one light source immuminating your scene, and everything cast shadows, and you rendered the shadows first, and black, as described above "this is crap" would not be visible, because that area would be in shadow because of the walls.

There probably isn't a way to render stencil shadows only for some things but not for others. If there is, I bet it requires the latest version of DirectX.

As for multiple light sources, I don't know how you do those. I suspect Doom didn't do them, cause I don't remember seeing anything other than single shadows in any area. I don't know how they could have done transitions between light sources though.

Hope that's of some help. My own shadow system does not stop at walls either. In order to do that, I would have to treat the projected shadow as a plane, clip all the polygons in the light's frustrum, sort them from front to back, and then clip away bits of that plane as I applied the shadow to each polygon from front to back until I ran out of shadow or polygons to clip it. This would be slow. Though probably not as alow as clipping all those polygons in the first place which I already have to do.

PS:
I could be wrong about that first bit. Maybe I am not seeing what I think I see in the first image cause of all your light sources. And maybe you can fix the problem with clever stencil shaodw stuff.


JoshK(Posted 2006) [#3]
I know Doom 3 uses a separate pass for each light, and adds the passes together. I want to avoid such a limited approach, because I want to render areas much bigger than Doom. I understand there will be some graphical glitches any other way, but I was hoping someone knew how games like FEAR combine stencil shadows with lightmapping. Maybe a clip plane method could be used to trim shadows off along the first face they hit.

I don't think the additive passes approach Doom 3 uses is practical for anything other than tiny corridors.


sswift(Posted 2006) [#4]
Well using clip planes had entered my mind, but that was why I said if you can it is probably in the latest version of DX. Also, I don't know much about them but I suspect that you'd have to set them up per poly, and that doesn't seem fast enough. I haven't played Fear. But are you sure it uses stencil shadows? Maybe it uses zbuffer shadows?


Tom(Posted 2006) [#5]
sswift: I think he is rendering correctly, the problem is he's not casting from the whole building, so it'll never look right with just the light being cast. If the light's static, then the shadow volume for the building need only be calculated once, so why not use stencil shadows for it anyway?

Also, clip planes work from DX7 onwards and work on pretty much every card out there that has DX7+ drivers. Make sure if you use carmacks reverse that you don't clip any shadow volumes else it screws up the shadows.

halo: In the first pic, are you using 4 lights to cast stencil shadows from?

Excuse my quick mock up of your bulding, but here you go (Blitz3D implementation):




sswift(Posted 2006) [#6]
Yeah, I was mistaken when looking at the frist pic, I was tired, and I thought I was seeing polygons casting individual shadows and overlapping as if you flattened a mesh on a plane and made it transparent. But he's just got multiple light sources.

So the only problem here is that he's not casting shadows from every object. Is there a way to avoid that? I don't know. In his example there, with the light casting a shadow onto the floor, the only way you make it look "correct" would be for there to be no shadow on the floor. So he would just want a shadow on the wall.

Maybe he could set a clipping plane to clip the object's shadow volume at some fixed distance from the object. But honestly, I can't think of ANY shadow system which would avoid this specific issue without clamping the distance at which the shadow can go. It would look the same in my shadow system, or even a zbuffer shadow system.

The only thing he could do maybe is right now he doesn't have shadow maps being rendered. If he rendered all his shadow maps black and the stencil shadows black in a first pass, and all the light sources were fixed in the same locations as they were to create the shadow maps, then the stencil shadows would blend with the shadow maps. It would look a bit funny with the sharp shadows and the blurrier shaow maps, but he'd get a scene somewhat similar to what you've shown, without the cost of rendering stencil shadows for nonmoving objects. His objects however would cast shadows onto eachother, but the world would not cast shadows onto them so there'd be a potential glitch there. And he would not want to use those stencils for that nonmoving light as in his example.


Tom(Posted 2006) [#7]
My system sets up an infinite far clip plane (via DX/DLL trickery) as per some tech paper on the subject, it stops the shadow volumes getting clipped even when extruded by 2147483647 units with a camera range of .1 to 100.

If I understand it all correctly, camNear to camFar is mapped to 0 to .9999 in the zbuffer, any verts further than the far clip plane, no matter how distant, are mapped from .9999 to 1, luckily you lose very little precision in the zbuffer.

The only drawback is all geometry falling within the view frustrum AND beyond 'camFarRange' will get drawn. You'd have to create your own clipper system or something.

Done properly, the only thing non-realistic about stencil shadows is the harsh edges and I doubt it's easy to seamlessly integrate them along with shadow maps.


JoshK(Posted 2006) [#8]
Well, there are probably a few tricks I could use to minimize visual glitches like this.

Doing away with lightmaps and going with the additive lighting passes method is a huge change. If the latest hardware can handle it, I'm okay with that, but it also changes my market a lot. If 3D World Studio 6 comes out, and it doesn't calculate lightmap textures, the indy market will have little use for it. Of course, if I can offer a state-of-the-art engine with a real-time editor, I gain a whole new market.


JoshK(Posted 2006) [#9]
Well, I won't get anywhere by holding back.

Uniform lighting system, here I come!


Tom(Posted 2006) [#10]
Good luck, and I hope you find some power-ups along the way :)


JoshK(Posted 2006) [#11]
God, what a mess. Unreal Engine 3 clearly uses lightmaps on some outdoor areas.


JoshK(Posted 2006) [#12]
Well, here it is with a single light and no ambient light.

Ambient light and other light sources can be added in additional passes.




Tom(Posted 2006) [#13]
Are you doing overlapping shadows from different lights? ( render for 'x' levels in stencil, 1,2,3 e.t.c)

That really eats the frame rate in my system.


JoshK(Posted 2006) [#14]
I don't understand your question.

The fake lighting techniques produce good results, but when I see a room full of stencil shadows, I know that it is just the right way to render lighting.


Tom(Posted 2006) [#15]
If your stencil buffer is >1bit and you render some geometry from 2 lights whos shadows overlap, then the stencil buffer should be set to 2 where they overlap.

If you're doing the 'render a dark transparent quad over the entire screen' method, you can set the stencil buffer to D3DCMP_EQUAL/GL_EQUAL, set the stencil reference to 1 and render. This will only draw where each lights casts shadows discreetly.

If you then set the reference value to 2 and rendered the quad again (with either a multiply mode or make the quad less transparent) it would only draw where the shadows overlapped.

With me now?

The other method I've not tried yet, where you render the shadow volumes first, then render the scene with just ambient where the stencil is 0, then normal lighting where the stencil >=1.