I'm no materials expert so I can't say for sure about the structural integrity of the tee nuts in there. Since they are cheap, I would just replace 'em. But you'll have a hard time finding stainless ones in metric. At least McMaster didn't have them in m5 last I checked.
And if you do add the washer, make sure you put a longer bolt in there. The factory ones, as I'm sure you know since you replaced yours, are m5x20mm.
->Ben