Germany dominated a frenetic match for large spells but failed to capitalise on their pressure - before Japan caused the Khalifa International Stadium to erupt into a mix of celebration and surprise.
Substitute Takuma Asano squeezed in a thumping strike from a tight angle to make it 2-1 to Japan and ran over to the photographers in the corner of the stadium in sheer delight.
There was a collective pause in the arena as the ball rolled down from the roof of the net before supporters realised it had gone in.
Manchester City's Ilkay Gundogan had given Germany the lead from the penalty spot in the first half, before Japan substitute Ritsu Doan equalised in the 75th minute.
The equaliser came after a string of stunning saves at each end - Japan's Shuichi Gonda pulling off a double stop to deny Serge Gnabry and Jonas Hofmann, before Germany's Manuel Neuer got his fingertips to Junya Ito's effort.
Germany, who finished bottom of their group in Russia in 2018, had missed several chances to extend their lead when the score was 1-0.
They face stiff opposition in Group E with 2010 winners Spain to come on Sunday, then a final group game against Costa Rica.
Before kick-off, Germany's players put their hands over their mouths during a team photo and in the stands a German government minister, Nancy Faeser, wore the OneLove armband.
It comes after news the German football federation (DFB) is investigating whether it is legal for Fifa to threaten players with sanction for wearing the OneLove armband in Qatar.
The DFB later tweeted that human rights are "non-negotiable" and "we stand by our position".
The misery of Germany's early exit from the World Cup in Russia could not be forgotten as Hansi Flick's side trudged off the field at full-time having suffered defeat at the hands of Asian opposition again.
Their supporters had arrived in hope more than expectation that their young stars would lead them into a new era.
Japan were relentless in energy and refused to give in, despite Germany controlling large spells.
There was a ripple of anxiety in the final moments from the Japan supporters at the opposite end to Gonda's goal as Germany's Leon Goretzka struck wide and goalkeeper Neuer came up for consecutive corners to try to salvage a dramatic equaliser.
However, Japan held on and, as the final whistle went, the players in the dugout raced on to the pitch to celebrate and flags were waved gleefully in the stands.
They had caused problems for Germany in the first half as Celtic forward Daizen Maeda had a goal disallowed for offside, before Germany's Kai Havertz also one chalked off for the same reason.
Gundogan and Antonio Rudiger both missed the target with decent chances in the first half, while midfielder Joshua Kimmich was denied twice by Gonda.
Gundogan coolly slotted his penalty down the middle after David Raum had been pushed down by Gonda to give Germany a deserved lead, but they were made to pay for their missed chances.
Bayern Munich teenager Jamal Musiala beat four players in the box only to blast an effort over the bar early in the second half, before Goretzka shot wide in the final moments.
-- Courtesy of BBC Sport