The Palmer House was built by Joseph Fish. Mr. Fish owned whaling ships. There is a ledgoned in falmouth that a pirate was kept in chains in the attic of 81 Palmer Avenue. A guest has reported that she heard chains rattling on one cold October night and a housekeeper has reported seeing him on the third floor of the Inn.