Allow me to swap out Party members without having to return to the Inn. Give the team a relic/amulet/treasure whatever that allows me to teleport team members in and out wherever I happen to be.
To keep this in check, give it a cost, like a Mana cost from the two remaining team members, or from the person coming in. In combat, make it cost a couple turns. One turn to select the action, one turn to “cast” it, and then the person that comes in starts at the end of the queue. Plenty of ways to keep a player from abusing it, but still allow me to be more strategic about my dungeon and combat game play.