Cam bolts are not suggested anyway. I will not use them on anything no matter what. Drop brackets are still the best for a DD and weekend wheeler (unless you are heavy in the rocks).
For a rough service trail rig and rock crawler adjustable control arms are best.

 

If I do any off roading it's dirt country roads. Definitely no rocks. So I'm trying to save on tire wear and help the ride. I've got a 14' JKU, 3" BDS suspension lift with JKS adj track bar, running 35" tires.

 

Ouch!

 

The cams have a small amount of adjustability. So I guess you could try and lower the caster with them, then the brackets would raise the caster back to where you want it? (theoretically, I haven't read about anyone doing this...)

The stock lower brackets were notched out when the cams were installed, and many people have complained about them slipping over time - especially with offroad use. Better to just remove them if you move to another form of correction. Do a quick search on 'cam bolt eliminator'. It's basically a set of thick washers to install when you remove the cams.