Should we stop hanging out during the week?

I've been hanging out with "Ken" for about a month now. I LOVE hanging out with Ken, he's really nice and GREAT in bed. The problem is, we have very different schedules. I work full-time, while he's unemployed. He lives with his mom and does not drive, so if we want to hang out, I have to go to his place to pick him up, which is about an hour's drive round-trip. I'm not really complaining -- the great sex more than makes up for the travel time lol. But the problem is, getting together during the week really wears me down. I have a sleep disorder he knows about. To make a long story short, I need 12 hours of sleep to get the equivalent 8 hours for a normal person. I know this isn't normal and I'm currently seeing a sleep doctor about it, but until I'm "cured", I'm going to be walking around like a zombie if I don't get 12 hours sleep. Yet he loves to hang out at my place during the week, and when he spends the night, I only get about an average of 4 hours sleep. I'm at work now, and I'm freakin EXHAUSTED. I can't remember the last time I've felt this bad. Should we stop hanging out during the week? I love spending time with him. Nothing makes me happier than cuddling with Ken on the couch. I even "topped" him a bit yesterday (which I've never done before because I'm a fem bottom boi lol), but I dunno if it's worth it if it means sleep-walking at work the next day. What do you guys think? Should we hang out during the week?