We have a similar story.
At our old house (which we rented) we suddenly had a cat in our garden, the other half actually though we had a snake. Turns out it was "kittys" tail. Sleeping on top of a planter in a mini greenhouse.
Cat started spending more and more time around us (I dislike cats) but we wern't feeding it. The the other half picked it up and said on my its weighs nothing. Me still not a cat person ignored this. Other half insisted I bought some cat food when shopping so I bought the cheapest tesco value biscuits. Took them home and it ate a massive amount, and how you imagine you would eat a prime steak if you had had nothing to eat for a week.
Over time this same cat slowly spent some time inside, I resisted this a lot, but she would let the cat in and eventually one day I was reading on the bed, cat hopped up and sat next to me. I am reminded now when other half got home saying oh shes quite sweet really.
The coming in got more and more and I kept saying we are not allowed a cat! Eventually we agreed to contact some cat places and when one had a space kitty was taken there. We were all actually sad, but thought it was for the best.
Next day we had a phone call, the cat belonged to our next door but one neighbours. Now these we knew we wernt that happy with. Had dogs locked in a small cage in the garden with them shock collars on to stop them barking.
But anyway said neighbours went and recovered kitty and she was back. The cat people said you have two choices if she keeps coming back, either ignore her or let her in and in effect adopt her. Next day she was back at our door, within 2-3 days she was clearly starving again and desperate to get in. That was it, she moved in and spent increasingly longer periods at ours.
The dilema happened a few years ago, we were moving to our new house, we no longer needed to be where we were (kids) so could go back to living away from town. We had to decide basically between taking kitty or not, we decided we had to.
Plenty of people have said cats choose their owners.
We waited around 6 months after moving to get her jabbed, we said we didnt think she was probably up to date and the vet said it would be fine to do the starter course on jabs again at that point.
The only thing we fear is that as shes chipped to the "old owners" should something bad happen they would get the call. Its just a risk we have to live with.
Shes a little bugger and I love her to bits, she literally converted me from being a dog lover to a cat lover.
So I would say, keep him, love him, and just accept your doing the best for him you can.