But having lost the likes of Fionn Carr and Ian Keatley to Munster and Sean Cronin to Leinster, Duffy admits only greater success will keep rising stars out West.
"It's up to us to move up the table and that's how we will keep hold of more players," said Duffy.
"We've great facilities. We've got everything on site, we've got a good gym and an astro-turf, the stadium has been developed, so everything is in place. It's just up to the team to progress.
"We're not talking about top four or winning the league or anything but we just need to keep improving, picking up more points and being more competitive every week and hopefully improve by one or two points next week.
"And if we do that people who come in will enjoy it, they'll enjoy living in Galway and they'll enjoy being part of the squad. It's only natural that some players want to go back to their own provinces. I'm a proud Connacht man and I don't know if I could play for any of the other provinces.
"You see the likes of Fionn Carr, Sean Cronin and Ian Keatley, they are all very proud of where they come from. Sean is going to Leinster and he obviously felt that was the best move for him. "It's just a shame that it's often the top guys but hopefully we'll get to the stage where they get to the end of a season and turn around and say I want to stay."