Muhyiddin Yassin is confident of gaining backing to become PM again
Opposition leader Anwar Ibrahim secured the most seats, but his Pakatan Harapan alliance fell well short of a majority.
The ruling coalition, led by Ismail Sabri Yaakob, suffered its worst electoral defeat ever, winning just 30 of the 178 seats it ran for.
Veteran politician Mahathir Mohamad lost his seat, suffering his first defeat in 53 years.
At 97, Mr Mahathir has dominated Malaysian politics for decades, first serving as the country's prime minister from 1981 to 2003, and he was credited for the country's rapid economic transformation from the 1980s.
The country is now however politically unstable, having seen three prime ministers since voters went to the polls in 2018.
The former ruling party UMNO suffered its worst ever election performance. Voters instead turned to Perikatan Nasional and the conservative Islamic party PAS.
Rival parties must now present the name of a candidate they think has the majority support in the lower house of parliament.
The stalemate means the backing of parties from eastern Malaysia, on the island of Borneo, will be essential, and they are expected to press for a better deal for their region under Malaysia's federal system.
A regional coalition, Gabungan Parti Sarawak, is willing to work with former premier Muhyiddin Yassin and the incumbent administration to form a new government, state media report.
"I am confident I will obtain enough support from lawmakers that will enable me to be appointed by the king as Prime Minister," Mr Muhyiddin said, without suggesting which other parties might back him.
-- Courtesy of BBC News