Workplace colleagues take loong breaks

I work retail part time and recently my colleagues have been taking longer and longer breaks. For about an hour everyday I am only one who is in store while others drink coffee and gossip. I am only male in my workplace. This is really bothering me but idk how to proceed as this is a mix of bossess and regular employees.

Was thinking about quitting on the spot (realistically I have a ton of money saved and I could do so), but am looking for another way that I can solve is. Thing is, I am not really good with confrontation and this gets me so pissed that I think it would devolve into a shouting match pretty soon.