If you want something more permanent than silicone seal, try Loctite Marine Epoxy, its approved for below the water line, so it should work.
I have not used it for filling rivet holes but I have used it on the hull, worked great. I got it at Lowe's it think it cost about 8.00, one tube should be plenty.
Lou