Gordon's Bay is the smallest of three towns in the Helderberg region, so named after the Helderberg Mountain which is part of the Hottentots-Holland Mountains which dwarf the locality on two sides. Gordon's Bay was originally named "Fish Hoek", many years before the town of the same name, located on the western side of False Bay, was founded. Evidence of this can be seen on the outside wall of the local Post Office.

Gordon's Bay consists of the old village, situated around the old harbour and Bikini Beach, the mountainside on the lower slopes of the Hottentots-Holland mountains overlooking False Bay, and the low-lying suburbs close to the main beach, making up the most recent expansion of the town.

The older part of Gordon's Bay is host to the South African Navy's Naval College, an officer training center located adjacent to the old harbour and yacht marina. Above the harbour on the mountainside is a giant anchor and the letters "GB". Many think that this stands for Gordon's Bay, but in fact it stands for General Botha which was the original name of the Naval station in the town. The Steenbras Dam, Cape Town's primary supply of drinking water, is located on the mountain above the town.