Definition: Keyhole may be a noun which refers to the hole which a tumbling bullet makes in a target, or it may be an adjective which describes the tumbling which the bullet was doing when it struck the target.
When a conical bullet strikes a surface in any attitude other than truly stabilized flight, that bullet has keyholed. Evidence of keyholing most often comes in the form of oblong holes in a paper target. Sometimes the holes are just a bit longer in one direction, and at other times the bullet will strike the target sideways. See the photo for examples of both.
The causes of keyholing may vary, but in all cases the bullet's flight was not properly stabilized when it struck the target.
In ideal conditions, a conical bullet is given some spin by the rifling in a gun's barrel as it travels through the barrel after being fired. That spin will stabilize the bullet's flight so that its nose is always forward, and it moves through the air in a nice straight manner.
If a gun's rifling is of the wrong twist rate for a particular bullet or velocity, if the rifling was damaged or absent, or if the bullet's diameter doesn't properly match the gun's bore diameter, the bullet is likely to tumble in flight. This results in poor accuracy, unpredictable bullet flight, and unpredictable bullet performance when it reaches the target or game.
Even a properly stabilized bullet may keyhole; this could be caused by it striking an object between the gun and the target.
Examples: The old Mauser's barrel was just shot out, as the keyholed bullets made clear.