Remove Shadows in Source Images

Originally written by ascottk & Renzatic on http://forums.thedarkmod.com/topic/4184

ascottk's Method

I used the GIMP for all of my texturing. Here's a quick & dirty tutorial for getting rid of shadows:

1. Copy the diffuse layer
2. invert the colors on the copy
3. overlay the copy on the original (experiment using overlay, multiply, grain merge, etc.)
4. adjust the layer opacity on the copied layer so it nearly cancels out the original
5. add a mask to the copy & get rid of anything that's totally cancelling out the details (the mask is white=opaque, black=transparent. use the airbrush with black or dark grey)
6. merge the two layers
7. adjust the gamma levels on the final image

Here's the best tutorial for tileable textures. I use this technique all of the time (the first texture on this thread looks like a mirror image of itself. It's obvious when you do the offset).

http://www.gimp.org/tutorials/Tileable_Textures/

Renzatic's Method

I have a somewhat simpler but more time consuming approach.

Convert your image from RGB to Lab Color Go to the channels tab and select the Lightness channel Dodge, burn, and curve your way to awesomeness.

If you do it right, you'll end up with a perfect image completely lacking in shadow information without any potential artifacts arising.

STiFU's Method

I use Photoshop CS3 for my texturework, but I am sure all functions and filters I use are also available in any other imageediting program, like GIMP. Please note that this is a very detailed description, with a couple of tricks, alternatives and workarounds. Not all of these steps may be necessary. The diffusemap is assumed to be tileable already.

Preprocessing

For the following steps to work properly, the brightness has to be distributed equally over your image, which also makes your diffusemap tile better, because less patterns are perceived. It is very common that an image is brighter on one side than on the other (e.g. top and bottom). There are various ways to fix this, here are three of them:

• Use the built in shading-/highlight-remover of Photoshop (Image->Adjustments->Shadow/Highlight). Play with the settings, until brightness is equally distributed over the image. If you set the radius setting to 0 px, shading will be removed pretty much completely. However, the result doesn't look good at all, which is why you should also read the rest of this article.
• Duplicate your diffuse-layer and add a layermask (click on the icon with the white circle in a gray rectangle at the bottom of the layerbrowser). Pick the Gradient Tool by holding on the "Paint Bucket Tool"-icon and chose a black to white gradient. With your layermask selected, drag a straight line from the bright side to the darker side to your image. If you hide the original diffuselayer now, you'll notice that your duplicate layer is only visible at the darker side and becomes more and more transparent at the bright side. Select the image of the duplicate layer in the layerbrowser and alter its brightness (Image->Adjustments->Brightness) and/or its exposure (Image->Adjustments->Exposure) to match the brightness of the bright part. Finally, merge both layers. Sometimes a colormatching is also necessary, which can be done easily with this method.
• Small brightness differences can also be reduced by the gammasettings (Image->Adjustments->Exposure).

Hint: If you use an offset (Filter->Other->Offset) with wrap-around, you can tell very easily when the brightness of bright and dark sides matches up.

Creating a layermask for the shading

First of I create a layermask for the shading, so that nothing but the shaded parts of the texture are altered later. As the shaded parts are a lot darker than the rest of the texture, I use thresholding (Image->Adjustments->Threshold) on a duplicate of the diffusemap to separate them from the rest. Play a little with the Threshold-level, until all shaded parts are within your layermask. You want to have a very clean segmentation of shaded and unshaded parts, but you will probably have some noise in your layermask now, which you eliminate using a medianfilter (Filter->Noise->Median). The advantage of a medianfilter over a blur is, that it maintains edges and eliminates noise more effectively. At last, invert the image and name the layer something convenient like "Texture XY shading layermask".

Creating texture for the shaded parts

Now we need texture that is applied to the shaded parts: The "shading-texture". You can use the inverted diffuse here, as ascottk suggested or use Renzatic's Method as well. However, I mostly use one of these three methods:

1. 

   Texture-offset example
   Depending on the structure of the diffuse, you can use the texture itself here, only slightly offset (with wrap-around), so that the shaded parts of the original diffuse are covered with non-shaded texture. In the roof texture on the right for example the shading is mostly only at the bottom of the shingles, so you can move the image down a couple of pixels. This way the shaded part of a shingle is covered by texture of the same shingle.
2. Use a very strong blur (Filter->Blur->Gaussian Blur) on the source image, so that the shaded borders aren't visible anymore and add noise (Filter->Noise->Add Noise) to the texture. If the shaded borders on your texture are rather narrow the noise will create an illusion of structure. If they are too wide though, the borders will look like what they are: A noisy blur.
3. Simply raise brightness/offset/exposure(/...) on a duplicate of your diffuse. The results of this method are not so good most of the time though.

Applying the layermask and final tweaking

Add a layermask on your shading texture, select it and apply the "Texture XY shading layermask" earlier created to it (Image->Apply Image, select the appropriate layer to use and make sure blending is set to normal). Now your shading-texture is only visible at the shaded-borders, but you will notice rough transitions between them and the rest of the texture. So select the layermask applied to your shading-texture and use Gaussian blur on it until you are happy with the result. You can also widen or narrow your layermask using maximum- and minimum-filters (Filter->Other->...), if you feel like the shading-border-size is not suitable yet. Furthermore you should adjust the opacity of this layer and maybe try other blending modes until you're satisfied.

If you used method one for creating the shading-texture, you might also perceive some repetitions of the original texture at the shading-edges. You can reduce those by using blur or a median on your shading-texture and adding noise afterwards.