But the League Two side were also given a helping hand by referee Trevor Kettle who left the Conference side's hopes of an upset in tatters.
After just 15 minutes, with the Stags already a goal up, he harshly sent off full-back Andy Sambrook and then just before half-time controversially gave the home side a penalty.
Jon Olav Hjelde had almost calmed Mansfield's nerves in the first minute, but the unmarked defender headed Stephen Dawson's cross well wide.
There was no let-off however for the underdogs in the 12th minute.
Mansfield forced two quick corners and from the second the ball eventually fell to Adam Rundle whose mis-hit volley fell kindly for Richie Barker to head in his 12th goal of the season from six yards.
Two minutes later the non-leaguers were left with a mountain to climb when Sambrook brought down Adam Rundle and was red carded for his late challenge.
But Grays nearly climbed that mountain within 10 minutes. John Martin's curling shot from a well-worked free-kick struck the foot of the post and with the goal at his mercy the unmarked John Nutter blazed the rebound over.
Four minutes before half-time Grays' hopes of an upset looked over. Stuart Thurgood clumsly barged over Adam Birchall and Barker converted the penalty at the second attempt after his first successful spot-kick was ruled out for encroachment.
Five minutes into the second half Barker squandered a golden chance to complete his hat-trick, firing wide of an empty net after a mis-kick by the keeper.
On the hour mark the striker again went close to his treble heading over at a Rundle corner after Allan Russell almost marked his full debut with a crucial thrid goal.
Russell again, Giles Coke and Brichall also wasted further chances to add to the margin of victory.
And they were almost made to pay 18 minutes from time when Nutter was only denied a goal by the finger-tips of Kevin Pressman.
That proved to be Grays' last hope as three minutes later Birchall glanced in a Rundle corner to earn Mansfield a place in the third round.