Do you still have the remains of the old clip? Can you still bend it back to an approximate shape?
If you fit the clip and handle without the plastic washer you should be able to convince yourself the clip will hold the handle.
I have found the best way to remove these handles is to lever the plastic washer away from the fabric with a large screwdriver (using a spanner on wood to protect the fabric) as if you are going to completely destroy the plastic washer in a rage, get hold of the clip and then heat the plastic washer later with a hot air gun before refitting it.
