Yes you are correct, and I installed the SD-A 5.5 Impellers. I didn't have a blade style feeler gauge but they are very close to the wear rings
Wear ring clearance is a big deal, using new BRP wear rings and a new or properly reconditioned factory impeller will result in a clearance tolerance of less then .010" in most cases. BRP says up .040 is acceptable as a wear guideline and both of those would look "really close" as you put it to the naked eye but I'd be looking for .010 or less and Nu-Jets are widely considered to be junk for having any quality control when it comes to tolerances. You could be off .080" and it would look close.
Your boat needs BRP# 271000497 impellers and they are an 11/22 pitch, throw the Nu-Jets in the scrap pile where they belong. Seriously, you're wasting time and money messing around with aftermarket crap that doesn't conform to necessary performance tolerances and specs. The Nu-Jets you used were designed for the 96HX jet ski and were never intended to run a boat hull even though they just happen to fit the pump. Get the OEM impellers for your boat, SBT has remans for 60 bucks a piece just send in your old cores to them.
Some of the other issues that can cause cavitation or not allowing the boat to get out of it's own way like you're experiencing are stripped driveshaft/pto splines, leaking ride shoes (you'll need someone with the puller to evaluate the boat for that). Also, the driveline can allow air to pass through the hull and into the pump if the bellows/carbon seal are worn and that might be part of your problem.
I'd take it all apart and start by pulling each drive shaft and making sure they are perfectly straight and all the splines are good at both ends and the pto hub splines on the engine are good. Have new rubber bumpers to replace in both ends of the shaft.
You might as well replace the entire shaft seal with new bellows, carbon seal, carrier o-rings etc while it's apart. SBT has a kit for doing all of that and it's much easier on the wallet than buying it all piece meal from BRP, I use lot's of them and they work well.
Use the correct impellers for your boat and inspect the blade tip clearance when you initially install them and while you're torquing the pumps to the ride shoe. Do whatever it takes to make sure the pumps are sealed to the face of the shoe, in your case using the neoprene seal was a good idea but it might help to apply a thin bead of silicone to the face of the shoe before the pump goes back on.