Rudiger is out of contract at the end of the season at Stamford Bridge after the Blues failed to sign him up to a new deal.
The Chelsea centre-back will now join La Liga giants Real Madrid on a free transfer once his contract expires at the end of June.
The Germany international has made 201 appearances since swapping Roma for Chelsea in a £29m deal in July 2017.
However, under Lampard – who is now Everton manager – Rudiger fell out of favour and Merson reckons that spell under the former England international made up his mind that he would leave Chelsea.
Former Arsenal midfielder Merson told Sky Sports: “Losing Rudiger to Real Madrid is going to break Chelsea’s heart – but his mind was made up 18 months ago with the way he was treated by Lampard.”
And Merson thinks Chelsea will now have to spend £70m to £80m to replace Rudiger and they won’t be able to find anyone as good as the German.
Merson added: “He got completely binned out the team by Lampard and I think that left a lasting effect on Rudiger. That’s when he decided that he’s not signing a new contract.
“Chelsea are letting a £70million-£80million defender go on a free transfer and they’re going to bring someone in for £70million who will be nowhere as good as Rudiger. They’re going to have to strengthen in defence as Rudiger and Andreas Christensen are going to leave.”
Chelsea lost the FA Cup final on penalties to Liverpool at the weekend and Rudiger was “frustrated” that he couldn’t leave the Blues fans with one last gift.
“It’s been five years as normal with ups and downs, but with a lot of positive things,” Rudiger said after the match.
“That’s why I’m even more frustrated because I wanted to have a different ending.
“At the end of the day it’s not only about one side, it’s both sides – Chelsea have been great to me, I have been great to Chelsea.
“As always, I am very, very thankful, me, my family. I became a man here. My kids were born here and everything. London and especially Chelsea will always be special to me.”