Oni's Coloring Tutorial

Welcome to this tutorial on how I color. I am using Photoshop 5.5 on Windows. The Mac key commands are different, so look in the help files to see what they are. I also use Photoshop CS sometimes, but I don't much like it. The brush pallet doesn't adequately suit my needs.

[edit: It's 2007 now, and while I think most of this stuff holds up, a few things have changed, so I'll make notes like these when something I originally had in here has changed over the years.]

New!: I now have a "Wet" tutorial up here.

This tutorial was done using a larger version of this PSD file, so feel free to use it for refference purposes: Winnowill-O.psd

 
Step 1: Setting up the Linework
First, the sketch. I load it up, use "Adjust Brightness/Contrast" to clear up some of the little smudges, and if it's very messy I tend to make a layer above it and paint over the problems with a white brush, erasing back the parts I don't need, and then compress the layer back down into one. I'll go into more detail about that technique later.
 
In this case, I then copy the background layer, and make that one a multiply layer. I then make a new layer I lable "white", which is just pure white. This is the new background and it sets the linework as an overlay, like cel painting. This ensures a tight border between line and color. I intend to remove the linework entirely on this one by the time I'm done, using it only as reference.
 
If I intended to keep the linework, then I would use a different method, which is to make the image RGB, drag one of the Channels in the Channel Pallet to the "new channel" button to duplicate it, then create a new layer called "lines", go up to "Select>Load Selection", and loading the apha channel. You might have to invert the selection (crtl-I). I then fill that layer with black, which creates lines JUST where there were lines in the sketch, and true transparency elsewhere. The benefit here is that you can create another layer above this one, group it, and then paint on it, resulting in "Disney-esc" colored linework (as seen in my Natalia kitten pic).
 
The third technique I use is to use the pen tool to digitally ink the pic, basically tracing over all my linework on a fresh layer, which results in very clean lines, but is a lot of work. This does make flats easier to do though, since you can just select the area you want to paint and paint bucket it without artifacts.
 
Since I'll be going without linework in the final draft, I'll make the linework layer mostly transparent, about 20% opacity, so that I can tell where the color edges are. While doing flats, I'll periodically hide the linework to make sure the color borders are smooth.
I generally work at a resolution of about 3K pixels to a side, give or take. This is to accomodate the size of the brushes I use.
 
Step 2: Flats
Ok, here's where I get into brushes. Here's a picture of the brushes I use: Many of them can be found in the Photoshop 5.5 "extras" dirrectory, but many of the rounded ones are my own blend. I use all the brushes up through the 300 soft, and then I made the 150 soft and the 400 soft. I also made up a 600 soft, and a 900 soft. Later, I made a 35 semi-soft (61%), 64, 45, and 150 hards (actually they range from 86-95% hardness to give them a softer alias), the 150 85% and 77% softness brushes (for large blocks of color with soft edges), and the 80, 60, and 25% hardness 65 pixel brushes. I use these a lot, I'll note when and where below. These are the brushes I've grown comfortable with, you might find a different mix better suits you. The other brushes in the set I use mainly for effects of various sorts.
 
So, flats. Step 1, above the white layer, I make a new layer called "skin". on this layer I paint in the areas with skin tone. since this is generally the bottom-most layer of color for the figure, I only worry about staying within the lines on the outer border. It can overlap other colors, like the eyes or hair. If this had linework, I'd use a very hard brush, and the linework would allow me to be a bit sloppier with it, since it covers up the actual edges. Without linework, it's important to get the borders very tight.
 
Anyways, when doing flats, I generally use one color, a bright orange. This color tends to contrast equally well with black and white, and with most other colors, so it's ideal for making sure the borders are sharp. Once I'm happy with the borders, I'll use Adjust Hue and Saturation to shift the color to what I actually want it to be. This can be found under Image>Adjust, or by hitting "crtl-U".
 

I'm using the 65 pixel, 80% hardness brush for this, but I might use a softer brush, depending on the effect I want.

[edit: Nowadays I tend to just use 100% hardness brushes to lay in flats. It's less forgiving of jagged edges, but looks sharper. I also use the pen tool a lot to do long lines, such as her sides. It's much easier than trying to do perfect lines with the brush, but the brush is better for fine work.]

Once the lines form a full circle, you can fill the inside, then just go over the white lines to make it a complete block of color, just to save a little time.

 

You can go outside the lines, that's what the eraser is for. If you leave the border of what you want, you can just erase it back out (generally using the same brush. It's worth learning that "ctrl-b" switches to the brush tool, and "ctrl-e" switches to the eraser. I do this CONSTANTLY, paint, erase, paint, erase, etc., so I find those commands Essential. Also during this part, you can create new layers, if working on one part will mess up another. For example, here I have the arm part on a different layer than the body, allowing me to paint in the arm without worrying about messing up the body part.

Once I'm done all the skin, I'll merge all those layers back into one. The key command for "new layer" is "ctrl-shift-n", and can also be found by hitting the folded paper at the bottom of the layers pallet. The command for "merge down" is "ctrl-e", and this will also "merge linked" if you link layers (by clicking on the empty box between the "eye" and the layer thumbnail on the pallet. Get used to all that, it's pretty common the way I work.

 
Note that the lines get very sloppy all around the hair (like on her right side). This is because the hair layer will go on top, so why bother stressing about something that'll be covered up anyways? I do want to clean some of this up though, so we get to our next tool, the Layer Mask. This is the little square with the white circle at the bottom of the layer pallet. With the skin layer selected, I click on this. Now there is a layer mask. I can erase on this all I want, but if I paint on it, all I can paint is what I've already erased. This is very useful for touch-up work, and for complex shading (which I'll get into later). Once you have the layer how you want it, click and drag the mask (the box next to the layer thumbnail) into the layer pallet trash can This will ask whether you want to apply, discard or cancel, hit "apply" (or discard if you don't like what you did).
 
Also learn to love "ctrl-Z", or "undo". This is what separates us from the cavemen.
 
The history is also good to use, when you made a mistake more than one move ago.
 
 
This is the finished skin layer before and after I adjust the hue and saturation:

What I did was reduce the hue a bit (making it more red), and lighten it considerably. I think this is a solid base skintone, but I might adjust it again later.

 
I'll make the "hair" layer next, since it's the next largest, but I'll sandwich some more layers in between later.
 
This is the finished hair flats, with the hue and saturation altered to fit the final color. I did use the pen tool to make some of the more sweeping hairs there. Over the course of making this layer, I made and collapsed about 5-10 layers, eventually ending up with the one Hair layer
 
Next I'd do a "clothes" layer, which covers the largest blocks of clothing, but she doesn't have any, so I'll skip that part. ;)
 
Note this part: I ignore the shoulder edge and ear for now. I'll go back in with a layer mask to cut those parts out in a second, but you don't need to get everything perfect on the first pass, trying to do so will just slow you down.
 
 
.
 

So next are the "ex" layers. I generally have three of these, and they are for the smaller blocks of color that go between the skin and the hair. The goal is to not have any two blocks of different colors being too close together, so that when I'm shading them I don't get any bleed over. Generally, the whites of the eyes, nipples, nails, and the teeth all go on ex1. Irises and lips generally go on ex2, eyebrows tend to go on ex3. Other stuff goes on whatever layer is relatively "clean" around them. Things like "eyelashes" and "eyeshadow" tend to go on their own layers.

[edit: Nowadays, I tend to put lips and nipples on a layer I call "m", grouped right in with the skin layers as a Multiply layer. I do this because it makes the transition between them and the skin tone more smooth. I do leave them on the ex layers if the lipstick is particularly strong (like a deep red), or if the nipples are a different color-range than the skin (like blue maybe).]

 
I should also take this opportunity to point out another trick, which I used to match the eyebrows to the hair. Once I'd gotten the shape of the eyebrows to my liking, I hit "/". This makes the layer so that whatever you do, the transparency remains the same. You can also access this feature by clicking "Preserve Transparency" at the top of the layers pallet. With this on, I simply selected the color of the hair, and sloppily paintbrushed it in over the eyebrows, but since the transparency was preserved, the shape of them was unchanged.
 

One other thing I'll point out is the "x" key, which swaps your foreground and background paint colors. Generally, when laying down flats, I try to keep one color as the orange I use, and the other as whatever color I might be sampling, in this case the black of the hair. I use "x" to switch which is the foreground color, dropping the orange to the back when picking and painting the black, and then hitting it again to swap them back.

When I get to shading, I tend to leave one color as the "s1" color I'm using, and the other as the "s2" color, but more on that in the next section.

 
Also, it should be noted that holding "alt" while the paintbrush is your tool changes the cursor into the eye drop (color picker). I also have gotten quite used to using "space" to move around, "ctrl-space" to zoom in, and "alt-space" to zoom out. These are all ways to increase efficiency. While I'm at it, remember to use "crtl-x, c, and v" respectively to cut, copy and paste, and to use "ctrl-t" for free transform. Very helpful.
 
Here's with the ex levels completed. At this point I decided that I didn't much like the shape of the left breast, so I moved the nipple a bit. Expect it to change a bit more over the shading process. ;) Also note that I used a softer edged brush to color the nipples. Nippes come in all shapes and sizes, I tend to draw them with softer edges, but they can have very hard edges, or almost no borders, it depends on tastes. I'm going to put nails on her too, but I've decided to wait until I've got the fingers shaded.
 
So that's the Flats. Congratulations for reading this far. ;)
Part 3: Shading