Date’s joke made me feel weird

Matched this guy on hinge, we talked for a few days then went on a date, it went very well and we both had a great time, a lot of good chats and laughters. Because of our work and travel schedules, we will both be busy the next couple weeks so on Monday (Veterans Day) which he had it off but I didn’t, he came over to my town to meet for lunch which was very sweet. Lunch once again went very well. On our drive back from the restaurant, I was telling him that I’ve been taking swim lessons and I joked about that I struggled to float, he all of a sudden said maybe it’s because you’re full of sh*t. It was obviously in a joking way and as soon as he said it, he realized it was inappropriate and apologized. I tried to shake it off and said it’s ok cuz he was just joking but I also felt weird , like there are so many ways he could have responded in a light hearted way than calling me full of shit…if we were dating for a while maybe it’d be ok but that was only our second date and we are basically strangers, which I would think to keep things polite and respectful.

Did he go too far on the joke ?