The factory axles have 6* of caster built in. So, at pinon angle 0*, caster is 6*. If pinion is 2*, then caster is 4* (6-2=4). That is about where you want to be. Somewhere between 4-5* is where we modified folks lifted that high typically are. As nuthinuf noted earlier, from the factory caster 4.2*.

Keep in mind that the axles aren't dead nuts perfect all the time from the factory, so there is a chance that one side's cam bolts are set a little different than the other, but likely it slipped as you note.