So, after looking into the bolt issue, it turns out that the bolts were loose, and I would have been the moron that didn't torque them, but after torquing them, and replacing the other ball joint, the shimmy is history and it is back driving like it should. I did order the caster adjustment...