Their narrow defeat in France means they are yet to win a game in Europe this season, and sit bottom of Pool Four with just two points.
Ruaridh Jackson missed the chance to win Sunday's game when his late penalty struck the post, meaning the Warriors slipped to their fourth consecutive defeat.
And Townsend believes it is just small margins that are costing the Glasgow outfit after watching his side surrender an 8-3 half-time lead.
"We have had two close games against Castres and we need to put those nearly wins into victories," he said.
"We expected a tough match. Castres are fourth in the French league and have an excellent home record and I think that is seven wins in a row they have put together.
"We knew how hard it would be but we believed in the team we had and we put them under pressure."
But the Warriors paid the price for ill-discipline as Tom Ryder and Byron McGuigan were yellow-carded, while Castres scored seven unanswered second-half points to secure the win.
And 39-year-old Townsend believes his side need to be more disciplined in future if they are to mount a charge for a RaboDirect PRO12 play-off place.
"Our scrum came under pressure, especially when we went down to 14 men. At the end of the day what killed us was two yellow cards," added Townsend.
"It is up to individual responsibility and discipline. I thought on a couple of occasions in the first half we had the choice to get back and defend or go for the ball that was verging on offside.
"I believed we would improve in the second half but when you are playing away in France and the crowd really get behind their team then you have to have emotional control and we didn't in the second half."
63x46.png){kind=logo}