You can't judge a cassette or chainring by a quick look at one tooth. The clearest signs come from how the bike behaves, and from the shape of the teeth when you compare them closely. Here's what the references describe.
What a worn tooth looks like
A cassette is the stack of rear cogs. The chainring is the toothed ring at the pedals. Both use the same tooth idea, and both wear where the chain pushes on them.
As a chain and sprocket wear together, the teeth become sloped at the back. The rollers then ride up the slope.1
On sprockets with taller teeth, the backs of the teeth can become hook-shaped rather than only sloped.1 Those are the two shapes to look for: a slope, or a hook.
Skipping
Skipping is the classic sign. Under pedaling load, the chain slides up over the tops of the sprocket teeth and jumps ahead to the next tooth.2 In extreme cases, the chain may lift entirely off the sprocket and skip forward.1
There's a trick to watch for. A new chain on a hooked sprocket may behave well during a spin on the work stand, but jump forward under power.1 So a quiet test spin doesn't clear a cog.
The skipping may also show up on only some cogs. The reference says this can happen with only some sprockets and not others, because they may have unequal wear.1
Clunking
One reference describes a particular sound. Where the chain is about to engage at the bottom of a hooked sprocket, a roller sits on top of a tooth. If the lower run is slack, that roller and the others behind it come around to the top sitting on top of the teeth. Then the chain jumps forward by one tooth with a clunk.1
The chainring is different
The chain meets a chainring differently from a rear cog. On the ring, jumping forward happens at the lightly tensioned return run of chain, and it involves less power loss.1
Wikipedia says that once a chain has worn past its limit, the rear sprockets are also likely to wear, in extreme cases followed by the front chainrings.2
Why a new chain doesn't always fix it
If the chain was left too long, the skipping can carry on after a new chain goes on, because the teeth have become unevenly worn.2 Another reference says that with a new chain and badly worn sprockets, the new chain will almost certainly skip.1
That's why people measure the chain first. The ruler check gives a direct measure of chain wear and an indirect measure of sprocket wear. At 1/16 inch past the mark, the source says the sprockets are probably undamaged.1 The ruler and gauge steps are on the measuring page.
What this page can't tell you
The saved references describe skipping, sloped or hooked teeth, and the clunk above. They don't give a way to judge a cog by sight alone, or a set of noises that prove wear. If you're unsure, have a shop look, and follow your own bike's manual.
For the next question, whether the chain can be replaced alone, see why chain, cassette and chainring often wear together.
Notes and sources
- Bicycle Chain and Sprocket Engagement and Wear, Sheldon Brown
- Bicycle chain, Wikipedia