The feature sits in the Rembrandt impact basin, the second-largest impact scar on the planet. The basin was discovered during Messenger's second flyby of the planet on 6 October 2008, a maneuver that allowed the probe to photograph 30% of the planet's surface not previously seen by spacecraft.
By examining the craters that formed on top of it, researchers estimate that Rembrandt formed in an impact some 3.9 billion years ago, near the end of a barrage of impacts in the inner solar system known as the Late Heavy Bombardment.
The impact that created Rembrandt also fractured the crust beneath it, allowing magma to flow to the surface and partly fill the 700-kilometre-wide basin.
But researchers cannot yet explain some of the features etched in that volcanic material: a spoke-like pattern of troughs and ridges emanating from the centre of the basin.
The pattern is even stranger than a mysterious spider-shaped pattern of troughs found in Mercury's Caloris basin, during Messenger's first Mercury flyby in January 2008, says team member Thomas Watters of the Smithsonian Institution.
Troughs and ridges, which are thought to form through very different processes, are not expected to be found lying side by side.
So-called "wrinkle ridges" are caused when the crust compresses, while troughs are formed when it is stretched, causing the surface to separate.
"What's so bizarre is these features are sitting beside each other. We've never seen anything like that – not in Caloris, not anywhere," Watters told New Scientist.
Models cannot yet explain how this feature might have formed, Watters says. Multiple episodes of volcanic material bubbling up from below may be needed to explain the features.
A 1000-kilometre-long cliff, or scarp, though to be formed as Mercury's surface, was found, cutting across the rim and floor of the Rembrandt basin. It is the longest such scarp to be discovered on Mercury.
Messenger will make one last flyby of the planet on 29 September before entering into orbit around Mercury on 18 March 2011, where it is expected to operate for at least one year.