MASKBLEND doesn't work at the same time as SHADEBLEND, but there is a way to make it work...
Here is some code from glmax2d.mod outlining how it works out what to do in the various blend modes:
Case MASKBLEND
glDisable GL_BLEND
glEnable GL_ALPHA_TEST
glAlphaFunc GL_GEQUAL,.5
Case SOLIDBLEND
glDisable GL_BLEND
glDisable GL_ALPHA_TEST
Case ALPHABLEND
glEnable GL_BLEND
glBlendFunc GL_SRC_ALPHA,GL_ONE_MINUS_SRC_ALPHA
glDisable GL_ALPHA_TEST
Case LIGHTBLEND
glEnable GL_BLEND
glBlendFunc GL_SRC_ALPHA,GL_ONE
glDisable GL_ALPHA_TEST
Case SHADEBLEND
glEnable GL_BLEND
glBlendFunc GL_DST_COLOR,GL_ZERO
glDisable GL_ALPHA_TEST
Default
glDisable GL_BLEND
glDisable GL_ALPHA_TEST
End Select
In masked blend, your sprite's alpha channel values have to be 128 or higher (0.5 to 1.0) in order to be drawn. Blending is also switched off so it will absolutely not do shadeblend or lightblend or alphablend at the same time.
Alphablend multiplies the sprite's RGB values by its alpha channel values, and then adds that to the backbuffer's RGB values multiplied by the inverse of the sprite's alpha values. In alphablend you can't do lightblend or shadeblend or maskblend at the same time. To get a pixel to not be drawn from the sprite, it doesn't matter what color the pixel is so long as the alpha channel has a 0 in it.
In lightblend mode, the sprite's RGB values are multiplied by its alpha channel values - similar to alphablend, but then instead of inversely scaling the backbuffer's values, it simply adds it to the backbuffer's values - which is why the colors get lighter and lighter until reaching white. You can't do alphablend or shadeblend or maskblend while you're doing lightblend. However, you can fake masking in lightblend by setting the `do not draw` pixels to black with 0 for the alpha value - it will end up adding 0 to the destination pixel, thus not changing it, thus making it seem like you're masking at the same time. But you can't have alpha's of 255 or 128 and expect it to do that.
Shadeblend multiplies the sprite's RGBA values (including the alpha channel), by the values in the backbuffer pixels. It doesn't do any further addition, and merely replaces the backbuffer pixel with the result of this multiplication. It won't do lightblend or alphablend or maskblend at the same time. However, a white pixel in your sprite will appear to have no effect on changing the background pixels because it will be multiplying 1.0 x 1.0, which leaves the pixel as it was - giving the impression of masking. So pixels you don't want to draw with shadeblend you should set to white - it doesn't matter what the alpha values are because they aren't used.
Shadeblend always multiplies EVERY pixel, black or otherwise. But if your pixel is black, it should not change the background color.
| 
