The game is a sequel to the Sonic the Hedgehog 2 for the same system and follows on directly from the end of the game, in which Sonic defeated his enemy, Dr. Robotnik; crash-landing on a floating island, Sonic encounters new character Knuckles the Echidna, and must once more retrieve the Chaos Emeralds while also working to stop Dr. Robotnik from relaunching his ship, the Death Egg.

After Sonic defeats Dr. Robotnik at the end of Sonic the Hedgehog 2, his space station, the Death Egg, crash-lands on a mystical floating landmass called Angel Island (or simply "the Floating Island" as it's called in the game manual). As Dr. Robotnik begins to repair the damaged station, he meets up with Knuckles the Echidna, the last surviving member of an ancient civilization of Echidnas that once inhabited the island. He is also the guardian of the Master Emerald, which grants the island its levitation powers.

Knowing Sonic and Tails will try to track him down, and realizing he can use the Master Emerald to power the ship, Dr. Robotnik dupes Knuckles into believing Sonic is trying to steal his Emerald. Shortly after, Sonic and Tails in their biplane, the Tornado, are in search of Dr. Robotnik. Sonic, possessing the emeralds from the events of Sonic 2, then turns into Super Sonic. As soon as they arrive, Knuckles ambushes Sonic from underground and knocks the Chaos Emeralds from him, returning him to normal. Knuckles steals the Emeralds and disappears inland. As Sonic and Tails travel through the levels, they encounter Knuckles in almost every level, hindering their progress.

In the last level, the Launch Base Zone, the Death Egg launches off for the second time, knocking Knuckles off a pole and sending him plummeting into the water. Sonic travels to a platform on the Death Egg, fights, and defeats Robotnik for the last time. The Death Egg is seen damaged and falling after Robotnik's defeat, after which it crashes back onto Angel Island- leading into the second half of the story in Sonic & Knuckles.