How do I make my 6yo sleep through the night?

My son(6) will not sleep through the night. I have managed to teach him to fall asleep in his own bed by himself but he doesn’t stay there! It’s driving me nuts.

Most nights, he tries to climb in bed with me. I had to put a stop to this due to him being very fidgety and preventing me from getting good sleep. So I put a futon in my room and (most) nights he will go to sleep on that (after waking in the middle of the night). Sometimes though, he insists on being in my bed. If I try to hold the boundary firm, it turns into an hour of screaming and crying until I break down and just let him in the bed out of sheer exhaustion.

But there’s also nights where he wakes before I’m ready for bed. My room is my space and I don’t always want my son in there because I’m doing adult things or just having alone time. If I try to take him back to his room, he either demands I stay until he’s fallen asleep(which takes ages, like 30+minutes) or screams and cries to stay in my room.

These two scenarios happen almost every single night. I don’t know what to do help it. I feel awful if I make him stay in his room while he’s crying. But I also feel awful for giving into him and either giving up my sleep or giving up my space. It’s a lose lose situation. Please give me advice and tips 😭😭😭😭