You literally have no remove party members button. I feel like the game is bricked now really. Unless i am missing something i cant talk to rylan to go fight the pirates.
Talk to said party members and ask them to stay behind