In the Village, Raylan would have throw himself onto the fire if it would save Athena and that hadn't changed with the change of scenery. He still remembers tucking a very drunk young girl under his arm and making sure she got home safely. He remembers finding her coming out of the tunnels, terrified and panicky. It was everything for him to keep her away from more of the same or worse.

So he's relieved when she agrees, no matter the amount of dissatisfaction he was witnessing. She could be dissatisfied. She would at least be alive to do so.

"Thank you. Now that you're up, you.. wanna brush your teeth or anythin' before you go?" Personally, he always had to piss when he first got up and Ava's house was about 15 minutes away. If they weren't going to sneak out, he could at least make sure Athena was comfortable as she could be.