The win means Munster are sitting third in the RaboDirect PRO12 standings going into the new year, but are still ten points behind Mark Anscombe's Ulster.
Munster's 24-10 victory at Thomond Park was Ulster's first league defeat so far this season, and was a fourth win in their last five clashes for Penney's men.
And although the Red Army secured the relatively comfortable win against a much-changed Ulster, Penney believes his side should have been better.
"We made some errors, we got penalised for a few things. We had the momentum and then suddenly it swung a little bit," Penney told the Belfast News Letter.
"Ulster may only have been in our half maybe three times in the second half and they scored on the back of a couple of penalties with a quick tap which caught us unaware. I was a bit disappointed about that.
"If you had offered me four points before the game, I would have, given the amount of people they rested, probably begrudgingly taken that.
"They have given a wider group of guys an opportunity now. It shows the depth they have up there, because tonight they competed pretty well across the park.
"We have won our last two games now, but I would like to think that we would be a little more ruthless if you like."
Guinness PRO12
Suite 208, Alexandra House,
The Sweepstakes
Ballsbridge, Dublin 4, Ireland