Do I need to keep my passport on me at all times?

Not always, often your passport is kept by the cruise line to streamline immigration and customs procedures.  You are normally advised when to take your passport ashore depending on the port of call.  Your key-card acts as a pass to get on and off the ship to ease the responsibility of having to carry your passport with you.