The reason he was dumped to begin with was that he once worked at a recycling plant, but his boss made cutbacks on him due to being unable to pay much. For this, an angry Libby dumped him, and upon meeting the three LKHFF teams, he persevered in trying to have Rae replace Libby as his new girlfriend, but she always refused.
Of course, he does get Rae to dance with him near the climax.
At the end, he begins, upon Rae telling him, to understand that she is waiting for the right man to come to him, and then Libby comes to apologize to Harvey for dumping him and they become boyfriend and girlfriend again.
Even when swimming, Harvey is always shown wearing shoes because of the toes on one of his feet having been amputated (though it's never shown, especially to the cubs to whom it's never told either).