A passport is not required for entry into a US territory from the US or another US territory. You are required to show government ID which shows you are a US citizen. Of course a passport is the best way to do that but other documents work as well. St. Croix Passport Requirements - Entry Documents - Necessary ID - US Virgin Islands - USVI
Not true. When I traveled to St. John and St. Thomas I could not get through TSA, get a bording pass, nor could I board without a passport. Territories of the United States are not the United States. Don't know about Puerto Rico but I would guess a passport is required to fly there as well. I'll chack this thursday with someone that traveled there a couple weeks ago and post the answer friday.