Ben White scored the second own goal of his professional career to hand West Ham the lead against Arsenal
Arteta made six changes to his starting XI, with Declan Rice, Bukayo Saka and captain Martin Odegaard on the bench.
The Arsenal side selected produced a below-par display - for which Arteta took the blame.
"I'm very disappointed. I'm responsible for that," he said.
"We're out of the Cup, we wanted to play a very different game and compete.
"The game took a direction because of the first goal but we have to see much more from the team and earn the right to win.
"I'm disappointed with myself. We wanted to play in a different way and we weren't able to do that. Every time we lose the pain is there.
"We have to use this pain and this defeat to prepare the best way for Newcastle [in the Premier League] on Saturday."
West Ham took the lead via an own goal in the 16th minute, with Jarrod Bowen swinging a corner in close to goalkeeper Aaron Ramsdale, before Ben White nodded it past him.
Tomas Soucek had hold of Ramsdale's shirt, but with no VAR at this stage of the League Cup, the complaints of the England international - making his first club appearance since 27 September in the previous round - fell on deaf ears.
Arsenal dominated possession but failed to break through, and they were punished five minutes after the break when Mohammed Kudus beautifully trapped Nayef Aguerd's 50-yard cross-field pass before finishing low through the legs of Gabriel.
West Ham's win was safe shortly before the hour when Bowen collected a poor defensive header from White and fired in a volley from the edge of the area, a deflection off Gabriel's thigh wrongfooting Ramsdale.
Odegaard slotted into the bottom corner with the final kick of the game, but the Gunners' 30-year wait to win the League Cup goes on.
West Ham are into the quarter-finals for the third successive season, where they will face Liverpool away.
The major talking point in the build-up was the return of Rice to West Ham for the first time since joining Arsenal for £105 million in the summer.
Rice made almost 250 appearances for West Ham and captained them to a Europa Conference League triumph in June, and manager David Moyes called for the ex-Hammers midfielder to be "welcomed back with open arms".
That call was mostly heeded by the home fans, who gave Rice a strong ovation when he warmed up for the first time midway through the first half.
By that point his new club were already trailing and while Arsenal had started well - West Ham did not have a shot in the first half, despite taking the lead - they produced little end product.
Rice was sent on from the stacked bench after the second goal to a heady mix of boos and applause, with whistles of derision when his first ball forward was overhit for a goal kick.
Shortly after that Arsenal went 3-0 down, and not even the arrivals of Saka, Odegaard and Gabriel Martinelli could keep their cup campaign alive.
West Ham, meanwhile, were much improved from the side that lost their three previous matches, and had been labelled a "shambles" by their own forward, Michail Antonio.
Bowen told BBC Radio 5 Live: "We had to fight, we knew we were playing a top side but it was a really good performance compared to the last few [games].
"We have not been at the level, but we were always going to bounce back and tonight was the perfect way to do it."
-- Courtesy of BBC Sport