Mahathir, who is now the interim prime minister, said MPs would vote on Monday for a new head of government or face an early election if they failed to agree on a candidate.
The 94-year-old said King Sultan Abdullah Sultan Ahmad Shah failed to find a candidate to command a parliamentary majority after consulting political parties earlier this week.
The king has taken the unusual step of meeting all 222 MPs, instead of just the leaders, to gain an indication of whether a government can be formed.
The Malaysian king usually determines which party or coalition has a parliamentary majority. That bloc then chooses the prime minister.
The Monday vote suggested by Mahathir would allow all members of parliament to vote for a leader across party lines. Mahathir says he wants to lead a unity government drawing ministers from any party.
Anwar Ibrahim, 72, who has been the prime minister in waiting since the 1990s, said only the king had the constitutional authority to appoint a prime minister. The Conference of Rulers, which includes nine hereditary rulers, is due to meet at the palace today to discuss the matter.
The nine rulers are seen as guardians of Islamic and Malay tradition. They select one among themselves to become king under the rotating monarchy.
Yvonne Tew of Georgetown University in Washington DC said: “(I)t is not clear who commands a clear majority and what to do about that.
“It’s hugely unclear what is going to happen … We are in a situation where the options are: first Anwar manages to command a majority of parliament; second that someone else has a majority … or if no one manages to command an absolute majority parliament could be dissolved.”
The crisis was sparked by Mahathir’s refusal to set a date to hand over the premiership to Anwar as agreed in their deal ahead of the May 2018 general election.
An orderly handover from Mahathir to Anwar had been promised within two years.
Anwar’s Bersatu abandoned the Pakatan Harapan (Alliance of Hope) in a bid to form a new government.
Mahathir his rumoured to favour the economic minister Azmin Ali, Anwar’s deputy in their half of the coalition, to take the premiership.
Prime Minister Mahathir Mohamad. Picture credit: US Military