The world of private prosecutions is a very unpredictable. Remember the Steven Lawrence trial. If the Police are going to prosecute you for dangerous driving then they will have to start the proceedings within 14 days of the alleged offence. If the Police were aware of the part played by the sunken manhole cover then it will be in their report. If so, then ask them for a copy. You could then seek legal advice (look in Yellow Pages for a Solicitor who offers the first appointment free) and find out if you have a case against the local authority.
If the Police Officers at the time considered a prosecution likely or even possible, they would have either cautioned you at the scene or visited you at home and interviewed you under caution. It doesn't seem that they did, so in the extremely unlikely event that a Police prosecution came about you could ask that any comments made at the time be disregarded.
One of the difficulties with whiplash injuries, is that you can prove the injury exists, but not that it doesn't. If he is only after a relatively small amount, such as £2-3,000, the insurance company may well just pay, as a court case they win may well cost more than the compensation.
You would also need to know what if any medical assessment was made at the time, or very soon after. If he had whiplash diagnosed weeks or even months after the event, I don't see how he could attribute it to your crash. I'm sure this is very worrying, but as has been said by others, a) Leave it to the insurance company b) Don't say anything to anybody who knows the other person and c) Don't say anything to him. Refer him directly to the insurance company. It's their job. Oh, and d) Good Luck.