Koh Samet, including all of the good parts, is a national park and has an entry fee. Thais pay 40 baht, foreigners 400 baht - a classic example of Thai dual pricing. If your ferry arrives at the main pier and you take a songthaew to the beaches, there will be a stop at the main ticket checkpoint. If your ferry arrives at one of the beaches, an officer will collect the fee as you step out of the surf. Note that there is plenty of foot traffic in and out of the park to the 7-Eleven, ATM or other shops and restaurants and if you have no bags you can nonchalantly walk into the park without anyone checking your ticket. (There is a road via the temple, take that and you won't get near the checkpoint.

Bus
The 1st class bus from Bangkok's Eastern Bus Terminal (Ekamai) to Ban Phe usually takes just over 3 hours, costs 276 baht, and terminates opposite the ferry piers (there's no direct service to Ban Phe from Moh Chit - nearest alternative is to Rayong, from there you can take a Song Thaew to Ban Phe). If you take a regular bus from Pattaya or Sattahip, you'll need to take a songthaew or charter a tuk-tuk to the ferry piers. Minibus services go straight to the piers.

Air
The nearest airport fielding commercial flights is U-Tapao, just east of Sattahip - about 45 km from Ban Phe.

Ferries
Ferries from Ban Phe to Ko Samet take around 30 to 45 minutes. Only buy a one-way ticket, as there's no discount on round-trip tickets and you won't have to worry about losing it. The ticket sellers may also offer to sell you the national park ticket - don't bother.