Would it be bad to decline a job I already accepted?

So I've seen a lot of these posts but they all seem to be for the corporate world and most people seem to be of the opinion that you have to think of yourself, or that the company wouldn't think twice about doing the same to you so don't worry etc etc.

And that makes sense for a big corporation, but the job I accepted is for a charity. And I feel so awful about the thought of taking back my offer because I know they've spent money on the interview process and my uniform etc.

I havent started the job yet, but since accepting the offer last week I've been offered another job, in the same industry except not with a charity. They pay around the same so that's not an issue, it's just the 2nd job the hours/days are better for me. Also the first job I accepted is only temporary, the 2nd job is permanent.

I feel terrible, I wouldn't feel so bad if it was any other company but it's a charity, one I love, so I feel awful. Would it be terrible to decline it now? Should I just stick it out?