The Welsh club launched a devastating counter from their own half deep into stoppage time with captain Justin Tipuric going over to turn a draw into a 26-19 win for the home side.
Scrum-half Rhys Webb was also influential as a mesmerising weaving try from a line-out on the 22 dragged Ospreys back into contention from 7-0 down.
With the game poised at 19-all after 80 minutes, Ospreys poured forward opportunistically to steal the points and leave their head coach delighted.
"We never settle for a draw," said Holley, whose side have won both their matches this season.
"I remember us scoring a try against Edinburgh in Meadowbank in 2003, the 89th minute, Richie Pugh. I'll never forget it, and ever since then, we've never given up.
"Likewise, at 19-all I was a little bit worried we were going to concede because Edinburgh have been here twice and scored in the last play of the game.
"It doesn't make for a good blood pressure chart but that's the nature of professional rugby.
"The endeavour, and the resolve at the end to score right at the death shows what we are trying to build with this group of players, which is really satisfying."
After Webb's score, Dan Biggar and Greig Laidlaw traded four kicks each until Tipuric's winning try.
However, Holley was irritated that big mistakes nearly cost his team, such as when Richard Fussell misjudged the ball's bounce to let in Edinburgh winger Tim Visser, who put the Scottish side ahead.
"After the game you feel a lot of emotions" said the head coach.
"Relief is one obviously, but there is frustration at some aspects of our performance. What I can say is we did try to play a lot of rugby tonight.
"We're delighted to get the win, and credit to the boys, they stuck at it right to the end. Edinburgh made it very difficult for us.
"We did create a lot of line breaks and a lot of opportunities to score that we didn't finish off.
"To take the positives out of it, this group of boys showed a lot of resolve to come back at the end, we've got the win and we're top of the league after two games."
63x46.png){kind=logo}