strobo32, maybe your QD setup takes care of this? Would love to see come picts or even a video on how yours is setup/swapped.
I had similar problems with the barrel being loose in the upper until the barrel nut is tight. The quick detach barrel system made it worse. The part about the Cry Havoc QDB is you have to be careful about tightening the 3 screws that connect the barrel to the barrel plate. The space must be equal and have equal torque, 40 inch pounds. Yes, I bought the tool.
Installation video
https://cryhavoctac.com/video-66.html
Here is a video of switching barrels. The 22:50 mark shows a noob switching between barrels.
Using the bolt as alignment tool can help but it's not a fool proof method. I think the problem with my Max-11A1/15 upper was the clam shells were not quite aligned, which made it difficult to keep the barrel concentric. Despite the shell alignment pins, the barrel was a tad loose in the upper (up down/ side to side) until the barrel nut was tight. Maybe there was debris between the shell haves. I'm still not sure if that was the problem, but when I took the receiver completely apart, thoroughly cleaned it and put it back together, I was supper careful about barrel alignment. I know it made a difference because the barrel nut was easier to screw down using only finger strength. It always felt like the barrel nut was cross threading before. Whatever problems led to the light primer strikes, they disappeared.
I also tried Scott's aluminum foil trick and was about to order some .003 SS shim for the space between the receiver and the barrel collar. After aligning the upper. halves, the barrel hole was more evenly round. Whatever the issues were, it is running great now and I will never take the receiver apart again.