I recently bought a 2008 Rubicon, and the previous owner had mounted a winch on the front. I pulled the line out this evening to respool it cleanly, and it seems that there are a few spots where it's been pinched and come un-wound a bit. It hasn't frayed, as far as I can tell.

I'll post a picture as soon as I can figure out how to attach one.

In the meantime, how does one determine just how mangled a steel cable can be before it needs replacement?

 

If it's unwound or kinked I'd replace it.

Just my opinion of course, but when you need your winch you need it. I've seen lines fail from exactly what you speak of and it is not a good thing to be a part of.

I went with synthetic line on my winch when I replaced the steel cable.

 

I would also go synthetic for the safety reasons and the weight savings. I know they cost more but the safety aspect alone should be a no brainer. I really don't know why steel cable has not been made illegal for passenger car winches.

 

Wire Rope Replacement Criteria
The following criteria determine when a wire rope is no longer acceptable for service:
• 12 randomly distributed broken wires in one lay or four broken wires in one strand in one lay (A rope lay is that length of rope in which one strand makes one complete revolution about the core)
• . Wear of one-third the original diameter of outside individual wires
• Kinking, crushing, bird caging, or any other damage resulting in distortion of the rope structure
• Evidence of heat damage from any cause
• Reduction from nominal diameter greater than those listed in the following: Maximum allowable reduction Rope Diameter (inch) from Nominal Diameter (inch) Less than or equal to 5/161/64 More than 5/16 to 1/21/32
 Destroy, rather than discard, wire rope to be retired
IF IN DOUBT, REPLACE THE ROPE.
 Wire rope that is not destroyed might be used again by someone not aware of the hazard associated with that use. Destroying wire rope is best done by cutting it up into short pieces.

 

So, from a bit of googling that I've done, the third wrap from the right would be bird caged, and grounds for disposal, correct?

There are a few other spots like that throughout the line. If they are all close to the end, is it safe to cut the line and use the remaining cable? If so, is there a best practice for doing the cut and finishing the new end?

 

If you replace it, you might consider the synthetic rope as found on the Warn ZEON Platinum S.

 

That is bird caged but I've seen a lot worse than that on peoples winches, it would be a judgment call on how the rest of the cable looked.
Cutting it and running a shorter cable is not a bad thing. For every layer of wire wrap around the drum you will lose approx. 10% of your rated pulling power above the drum(10% per layer) so running a shorter cable gives you more pulling power to start. The down side is you will need cable extensions for long pulls.
Find a local club, someone might have cable just laying around.
Synthetic is expensive but worth it IMO