The Bees arrived at The Valley on the back of five successive victories, but fell behind after only three minutes and were three goals down in just over an hour.
But they battled back to score a goal and hit the post but it was not enough to repeat the success they had in the previous round when the knocked out Sunderland.
Brentford's manager Martin Allen was disappointed at his teams defeat and said he genuinely thought that they would win the match.
"We were only three games from Wembley and possible playing in Europe. It was tough, especially after conceding the goals at the wrong time. I was impressed by Charlton's attitude which was spot on otherwise we would have beaten them."
Charlton, who have a notoriously bad record in Cup competitions and have been knocked out on three occasions by teams from lower divisions, have reached the quarter-finals for only the third time since they won the Cup in 1946 and their manager Alan Curbishley is hoping that perhaps this could be their season.
"We have not had the best of times in Cup competitions over the years and this is the one area in my 15 years as manager that we have not done very well, so we will just have to wait and see what the draw brings."
Charlton stormed into an early lead when Darren Bent raced clear of the Brentford defence to fire home his 17th goal of the season and the strike was a big blow to Brentford who were cheered on by 5,500 of their supporters.
The visitors failed to test Charlton goalkeeper Thomas Myhre in the first 45 minutes and their only chance was a Lloyd Owusu header which went wide.
When it looked as though they had weathered the early storm, they conceded a second goal right on half-time, giving away a free-kick on the edge of the penalty area.
Charlton striker Jay Bothroyd, playing in place of the cup-tied Marcus Bent, crashed the ball left-footed into the net. It was only the fourth time he has scored such a goal this season.
With nothing to lose Brentford went for broke in the second half and Myhre had to make his first real save of the game, turning away an Owusu shot but it was the home side who scored the third and vital goal on the breakaway.
Substitute Shaun Bartlett, with almost his first touch of the ball, put Bryan Hughes clear and the midfielder slotted the ball home through the legs of Brentford goalkeeper Stuart Nelson to net the 15th FA Cup goal of his career - quite a record for a midfield player.
As Charlton took their foot of the pedal, Brentford peppered the home goal and were rewarded when substitute Isaiah Rankin hooked home a consolation goal for them and things could have become very interesting in the last ten minutes had Sam Sodje's header gone him instead of hitting the post.
The Brentford fans gave their team a standing ovation as they left the field and they will now be able to concentrate on promotion as they are currently well placed in a play-off position.