Sharif said the PPP had repeatedly broken promises on resolving the issue and on who should be the next president.
"We therefore feel that these repeated defaults and violations have forced us to withdraw our support from the ruling coalition and sit on the opposition benches," he said.
"We'll play the role of a constructive opposition."
"We don't want to be instrumental in overthrowing any government. We don't have any such intentions," he said.
Stability concerns
Farhatullah Babar, a PPP spokesman, welcomed Sharif's pledge.
"In a way, the coalition will remain intact," Babar said. "We still hope Mr Sharif will rejoin the coalition."
The move is not expected to force a general election as the PPP, the largest party in the coalition, still has enough allies in parliament to hold the government together.
But analysts say governing in the long term will be difficult with Sharif in opposition and the split is likely to spill over into Pakistan's upcoming presidential election, scheduled for September 6.
The PPP has nominated Asif Ali Zardari, its leader, as its presidential candidate, while Sharif has named a retired judge to run against Zardari.
Upcoming election
"We are now looking at a situation where there is going to be a potentially bruising presidential race," Mike Hanna, Al Jazeera's correspondent in Islamabad, said.
"You are looking at a situation where the electoral process is going to be divided ... a process that should have been the shining light and the final move towards re-eastablishing democracy is going to be a process that threatens to create more and more division within a country it was trying to unite."
The political bickering has added to Pakistan's woes and raised concerns about the country's stability at a time when the military is carrying out a crackdown on pro-Taliban and al-Qaeda fighters in the tribal regions.
Suicide bombings in the last week have killed nearly 100 people.
The Pakistani Taliban says the bombings were carried out in response to the military campaign against them and have threatened more attacks.
The government said on Monday that it had banned the main Taliban umbrella group Tehreek-e-Taliban Pakistan and frozen its bank accounts and assets in Pakistan.
Judges dispute
Musharraf sacked the judges last year, fearing they might mount a legal challenge to his re-election as president while he was also head of Pakistan's army.
But their sacking set off a wave of public protests and eventually led to the coalition's threat to impeach Musharraf before he stepped down on Monday last week - nine years after he came to power in a military coup.
While Sharif had demanded the reinstatement of the judges, the PPP refused to state when they would make a decision.
Analysts say the PPP is unwilling to restore the judges because of fears they could overturn an amnesty on corruption charges that allowed Zardari to return to the country last year.
The coaltion had been able to rally together against Musharraf, pushing him to quit to avoid impeachment charges.
But with Musharraf gone, the coalition has frayed.