Though I'm not nearly as knowledgeable as EuclideanCoffee in security matters, in answer to your question:

If it were me, yes, I would make new keys and nuke the old ones.

Although a bad actor...