I f/26 was visiting long distance bf m/26 and he got a weird text with no explanation

I was visiting my long distance boyfriend this past weekend and while we were out to eat he got a text from an unsaved number saying “why you ignoring me”. I didn’t mention it right away but he knew I saw it and I brought it up after we were done because it bothered me. He said he didn’t know who it was and that was as much explanation I got from him. It still was bothering me so I asked to see his phone and he told me no. He did the whole if you can’t trust me then this isn’t going to work thing and that he doesn’t ask to look through mine. I’ve never cared to look through his up until now.

I’m back home now and my gut feels really off by this and it’s still bothering me. We’ve been together 3 years but I feel like if I’m going to be questioning and not trusting that this isn’t worth it. I feel like if he didn’t have anything to hide letting me see wouldn’t be an issue. How would you proceed with this issue and/or the relationship?

EDIT: lol well hi everyone. I was just looking for a few opinions and didn’t think this many would have something to say. After reading through most of these I feel that I should clear a few things up and also add a little context. To begin, this was the first shady thing that happened while in person. Like I said before I never had a reason to want to check his phone before. I’m really the chill type and don’t care about phones until I have a reason to. Times we aren’t together tho some weird things have been happening the last few months that have bothered me and I’ve brought up to him. Not responding after a certain time of night, not updating me while he is out/gets home etc. We would talk about it and he would say he would work on it but then do the same thing. These might not matter to some but while we are LD these small things make me feel better. We have plans to finally move together when my lease is up in July so knowing these things now is important for obvious reasons.

Also, I noticed that some of you have noticed another post I made a few weeks ago regarding a miscarriage. This is still not easy for me as I’m still processing. It’s a lot to explain so feel free to go and read as that will explain a lot as well. Me going to see him this last weekend was the first time we were seeing each other after losing the baby. I was debating so hard even going but for the sake of our relationship I felt I should. Think of it as a last ditch effort.

I also messed up the ages on this one which I saw a few pointed out as well. He is 29 and I am turning 26 this week. I just figured I’d round up.. lol.

Last thing…. to those who do read my other post and ask or judge why I would stay know that I’m processing and dealing with everything that has happened on my own. This has been really hard for me and my head and heart have really been at war. I just ask you give some grace.