The ruling by the Constitutional Court on Thursday is the most serious setback for the AK party, which is headed by Recep Tayyip Erdogan, Turkey's prime minister, since it was elected in 2002 and may threaten its survival.
The defeated amendment is set to play a central role in a separate case that seeks to have the AK Party closed down over its alleged anti-secular activities and 71 of its members, including the prime minister and the president, banned from belonging to a political party for five years.
"This verdict will affect the closure case negatively," Mustafa Unal, a columnist for Zaman, a Turkish daily newspaper, said.
About 500 women demonstrated in the southeastern city of Diyarbakir after Friday prayers and hundreds more in colourful headscarves chanted slogans in Istanbul.
Thousands of women have not gone to university because of the ban -which has been strictly enforced since 1997 - or have gone abroad to study.
The headscarf reform has rekindled a decades-long dispute over the role of Islam in a country of 70 million that is officially secular but predominantly Muslim.
Turkey's secularist establishment, including army generals and judges, suspect the AK Party of harbouring an Islamic agenda.
Investment affected
The Constitutional Court is expected to rule in the coming months on the party closure case brought by the Court of Appeal's chief prosecutor.
Hurriyet, one of Turkey's leading newspapers, said the AK Party executive was likely to discuss the political crisis, including the possibility of calling an early parliamentary election.
The political uncertainty has hit the Turkish lira and bonds, as fears grow that the government will have to put its economic reforms on hold.
A note from Ekspres Invest, a Turkish brokerage, said: "The only certain fact is that given the essence of the turban [headscarf] verdict, political uncertainty will increase progressively in the months ahead until the political vacuum is filled."
New party
Many analysts expect the AK Party to be outlawed, although some say the court could instead decide to punish the party's leaders, given that forming a new political party would be easy under Turkish electoral law.
Senior AK Party members told the Reuters news agency that the party was considering the possibility it could be closed and Erdogan banned from politics.
The sources, who declined to be named, said the party had begun planning to create a new political party.
Turkey has a history of banning political parties and a predecessor to the AK Party was banned in 2001 for "Islamist activities".