Styling Replacing handbrake lever

[XYFAS]

I'd like to replace the handbrake lever cover with something more stylish than the plastic cover presently on it.

Any ideas how to remove it? I took the leather surround off but couldn't see an easy way to get the plastic outer off the leaver