People who won't accept your answer when they ask, "Where are you from?"

I am Canadian. I was born here, as were my parent, grandparents, great grandparents, etc.

So people ask me where I am from, I say here, then expand, I lived in another province for 20 years, but I was actually born in this town, and returned to the town in my mid 30's.

People always follow up with, but where is your family from and I explain, here. My mother was born in the same hospital I was. My grandmother born at home, but about 20 minutes away.

They then ask if I am native, and no, originally way back when, my family came from Ireland. So they say I am Irish, but my family came over during the potato famine. I am not Irish, no one in my family back beyond my great grandparents have ever been to Ireland. It's not really relevant, because it isn't our culture, or anything, because we have been Canadian for several generations.

I get asking where you are from to know if someone is local, or from the city, or to allow them to take that wherever they will, but also, if someone gives you an answer, not sure why you need to keep digging?

Edit for clarity: I am white, exceptionally white.