Ram Baran Yadav, an ethnic Madheshi from Nepal's lowland Terai area supported by the Nepali Congress party, was just four votes short of a majority, according to Nepal Television.
Ramraja Prasad Singh, the candidate backed by the Maoists and himself also a Madheshi, won 282 votes, the channel said.
Singh is also backed by three smaller parties representing the Madheshi.
The failure to elect a president delays efforts by former Maoist fighters, who hold the most seats in the constitutional assembly but do not have a majority, to form Nepal's first republican government.
Kul Bahadur Gurung, constituent assembly chairman, announced that another election for the president will be held again on Monday.
However there was one success on Saturday, the Madhesi People's Rights Forum managed to get their candidate, Parmanandra Jha, elected as vice president.
Political stalemate
Nepal has been without a fully-functioning government since elections in April, and the inability of Nepal's three main parties to reach a compromise has meant no budget has been issued for the coming year.
The country is also struggling with a series of strikes by workers over fuel prices, low wages and poor working conditions.
The assembly must still draft a new constitution, decide whether and how to merge the former Maoist fighters with the national army and oversee a human rights reconciliation process.
Girija Prasad Koirala, Nepal's interim prime minister, resigned in June but with no one in power to accept his resignation has lingered on as a caretaker.
Once a president is in place, a government is expected to be formed, headed by a prime minister, most likely Prachanda, the Maoist leader.