Should I let my cat's former owner see him?

This question is the heart that I was trying to get to with my last question, but my facetious approach didn't translate well online. Here is the gist, again. I adopted a cat in December. When the adoption happened, I thought it would be fine for the former owner to see the cat. She wants to do that now. The thing is this cat and I have not fully clicked. We just haven't fully gelled--nothing serious. I had a cat die soon before the adoption and Referee, the new cat, was rescued from the freeway from his former owner. We both were sad when the adoption happened. I think it is just taking time. I just don't want him confused by seeing his former owner OR more likely resenting me because if she is able to visit then he should be able to go home with her (in his mind). I have tried to explain these things to the former owner, but she doesn't seem to get it. I am sure she just wants to make sure that the cat is cared for. I would probably want the same. THANKS!