Remove Shadows in Source Images

Originally written by ascottk & Renzatic on http://forums.thedarkmod.com/topic/4184

General Information

A photo-texture is afflicted with shading of which there are basically two kinds: There is diffuse shading created by diffuse light, which is always present in a photo-texture and there's sometimes very distinct directional shading induced by lightsources like the sun. The latter has to be removed completely, so that the texture can be shaded according to the Normalmap of the texture and the present lights in a virtual scene, by the renderer of Doom 3.

The diffuse shading however should be kept the way it is, because it fortifies the illusion of depth and leads to a more realistic look of the texture. It will also look more realistic than an automatically generated Ambient-Occlusion-map that could be worked (baked) into the Diffusemap. A properly created AO-map might be more suitable for the structures of the created Normalmap though.

Because the directional shading is so pronounced, it can be separated from the rest, as described in STiFU's Method.

Nosslak's method

1. Open up your image in Photoshop (it might be possible in GIMP too but I use Photoshop so I don't know) and duplicate the layer with the bad lighting/shading.
2. Change the blend mode for the new layer from "Normal" to "Luminosity".
3. With the new layer selected apply the "High Pass" filter from the "Filter>Other" and experiment with the slider until it looks good.

This method doesn't always work (it might give you a little funky colors) so if you don't get any good results after this and some further experimentation on your own, just try the other methods.

ascottk's Method

I used the GIMP for all of my texturing. Here's a quick & dirty tutorial for getting rid of shadows:

1. Copy the diffuse layer
2. invert the colors on the copy
3. overlay the copy on the original (experiment using overlay, multiply, grain merge, etc.)
4. adjust the layer opacity on the copied layer so it nearly cancels out the original
5. add a mask to the copy & get rid of anything that's totally cancelling out the details (the mask is white=opaque, black=transparent. use the airbrush with black or dark grey)
6. merge the two layers
7. adjust the gamma levels on the final image

Here's the best tutorial for tileable textures. I use this technique all of the time (the first texture on this thread looks like a mirror image of itself. It's obvious when you do the offset).

http://www.gimp.org/tutorials/Tileable_Textures/

Renzatic's Method

I have a somewhat simpler but more time consuming approach.

Convert your image from RGB to Lab Color Go to the channels tab and select the Lightness channel Dodge, burn, and curve your way to awesomeness.

If you do it right, you'll end up with a perfect image completely lacking in shadow information without any potential artifacts arising.

STiFU's Method

The basic idea of this method is to apply a "new" texture only to the shaded parts of an image, so that the overall look&feel of a diffusemap isn't changed too much. I use Photoshop CS3 for my texturework, but I am sure all functions and filters I call are also available in any other image-editing program, like GIMP. Please note that this is a very detailed description, with a couple of tricks, alternatives and workarounds. Not all of these steps may be necessary.

Preprocessing

For the following steps to work properly, the brightness has to be distributed equally over your image, which also makes your diffusemap tile better, because less patterns are perceived. It is very common that an image is brighter on one side than on the other (e.g. top and bottom). There are various ways to fix this, here are three of them:

• Use the built in shading-/highlight-remover of Photoshop (Image->Adjustments->Shadow/Highlight). Play with the settings, until brightness is equally distributed over the image. If you set the radius setting to 0 px, shading will be removed pretty much completely. However, the result doesn't look good at all, which is why you should also read the rest of this article.
• Duplicate your diffuse-layer and add a layermask (click on the icon with the white circle in a gray rectangle at the bottom of the layerbrowser). Pick the Gradient Tool by holding on the "Paint Bucket Tool"-icon and chose a black to white gradient. With your layermask selected, drag a straight line from the bright side to the darker side to your image. If you hide the original diffuselayer now, you'll notice that your duplicate layer is only visible at the darker side and becomes more and more transparent at the bright side. Select the image of the duplicate layer in the layerbrowser and alter its brightness (Image->Adjustments->Brightness) and/or its exposure (Image->Adjustments->Exposure) to match the brightness of the bright part. Finally, merge both layers. Sometimes a colormatching is also necessary, which can be done easily with this method.
• Small brightness differences can also be reduced by the gammasettings (Image->Adjustments->Exposure).

Hint: If you use an offset (Filter->Other->Offset) with wrap-around, you can tell very easily when the brightness of bright and dark sides matches up.

Creating a layermask for the shading

First of I create a layermask for the shading, so that nothing but the shaded parts of the texture are altered later. As the shaded parts are a lot darker than the rest of the texture, I use thresholding (Image->Adjustments->Threshold) on a duplicate of the diffusemap to separate them from the rest. Play a little with the Threshold-level, until all shaded parts are within your layermask. You want to have a very clean segmentation of shaded and unshaded parts, but you will probably have some noise in your layermask now, which you eliminate using a medianfilter (Filter->Noise->Median). The advantage of a medianfilter over a blur is, that it maintains edges and eliminates noise more effectively. At last, invert the image and name the layer something convenient like "Texture XY shading layermask".

Creating texture for the shaded parts

Now we need texture that is applied to the shaded parts: The "shading-texture". You can use the inverted diffuse here, as ascottk suggested or use Renzatic's Method as well. However, I mostly use one of these three methods:

1. 

   Texture-offset example

   Depending on the structure of the diffuse, you can use the texture itself here, only slightly offset (with wrap-around), so that the shaded parts of the original diffuse are covered with non-shaded texture. In the roof texture on the right for example the shading is mostly only at the bottom of the shingles, so you can move the image down a couple of pixels. This way the shaded part of a shingle is covered by texture of the same shingle. If your texture is not yet tile-able, some additional work might be necessary here. But don't go ahead and hide the seam just yet, because only some parts of it will be visible later.
2. Use a very strong blur (Filter->Blur->Gaussian Blur) on the source image, so that the shaded borders aren't visible anymore and add noise (Filter->Noise->Add Noise) to the texture. If the shaded borders on your texture are rather narrow the noise will create an illusion of structure. You might want to widen the structures introduced by the noise, by applying maximum- or minimum-filters, but be careful here, because overdoing it will result in blocky structures. If the shaded borders are too wide though, they will look like what they are: A noisy blur.
3. Simply raise brightness/offset/exposure(/...) on a duplicate of your diffuse. The results of this method are not so good most of the time though.

Applying the layermask and final tweaking

Add a layermask on your shading texture, select it and apply the "Texture XY shading layermask" earlier created to it (Image->Apply Image, select the appropriate layer to use and make sure blending is set to normal). Now your shading-texture is only visible at the shaded-borders, but you will notice rough transitions between them and the rest of the texture. So select the layermask applied to your shading-texture and use Gaussian blur on it until you are happy with the result. You can also widen or narrow your layermask using maximum- and minimum-filters (Filter->Other->...), if you feel like the shading-border-size is not suitable yet. Furthermore you should adjust the opacity of this layer and maybe try other blending modes until you're satisfied.

If you used method one for creating the shading-texture, you might also perceive some repetitions of the original texture at the shading-edges. You can reduce those by using blur or a median on your shading-texture and adding noise afterwards. Again, you can widen the noise-structures, by using maximum- or minimum-filters.