When to physically escalate?

Hello fellow men,

Yesterday I went to the store, and on my way to the self-checkout machine, there was a guy kind of stumbling around. He was about the same age as me (mid to late 20s). Despite clearly wanting to walk forward, he sorta walked back as I was passing by him, and we bumped into each other.

Instinctively, I went "sorry, man" and kept walking to the self checkout.

He turned around and shouted "what 'sorry, man'? Puta." and some other swear words directed at me (that I didn't make out as I had my earbuds in). I could feel he wanted a fight.

He continued to swear at me as I calmly paid, took my stuff, and left. I did not think much of it and left. Nothing else happened.

Despite all this, by the time I got home, a part of me felt like a pussy in the sense that I should have stepped up. I acknowledge this is very much a primal, irrational train of thought, but I nevertheless feel like I should have spoken back at him, and if necessary, fight. I have done martial arts in the past, and given his skinny stature, I could probably take him; and hey, even if not, the primal part of me tells me that even if I got my ass beat, I at least "stood up" for myself.

I now have these two mindsets clashing in my brain, i.e.: a rational part telling me that walking away and not escalating was the correct, mature thing to do; vs. an irrational part telling me that I'm a pussy for not "standing up" for myself, which in practice means talking back to him and fighting him if it comes to it.

So my question, broadly, is: when is it OK to engage in this sort of tomfoolery and (be ready to) escalate? Is it ever? Or is it always better to walk away? Any other thoughts are appreciated too.