At least three UN vehicles were attacked and one set on fire as protesters broke doors and freed about 10 Serbs detained in the court raid, witnesses said.
Hundreds of UN forces later withdrew from the town after coming under attack, leaving Nato troops in place to restore order to the area.
Beshim Hoti, a spokesman for Kosovo police, said that UN police were targeted as they tried to arrest Serbs inside the courthouse.
"I suppose it was a hand grenade activated in the courthouse yard," he said.
At least one member of the international security forces was shot, a witness said, leading to an order for UN forces to withdraw.
Several injured
Polish national police said 22 officers on duty in Kosovo were injured in the clashes.
Witnesses said several demonstrators - lobbing stones and firebombs at police - were also injured.
Milan Ivanovic, the Mitrovica hospital director, said at least 100 people had been treated for the effects of tear gas.
More than 500 mainly Ukrainian UN police were involved in the dawn operation, Reuters news agency said.
Hajredin Kuqi, Kosovo's deputy prime minister, called for the peacekeepers to continue their operation against the protesters.
"We have requested from the first day that UN and K-For [the Nato force] establish the rule of law in north Mitrovica and to protect institutions there. It was a just action and the right one," he said.
Serb seizure
A group of about 300 Serb protesters opposed to Kosovo's independence took control the UN-run courthouse last Friday.
They refused to leave the premises after negotiations with UN officials failed over the weekend.
The Serbs - many of whom worked in Kosovo's judiciary before the territory came under the administration of the United Nations in 1999 - said they wanted to set up their own court.
Kosovo's government unilaterally declared independence from Serbia on February 17 and has since been recognised by many Western countries.
Serbia and Kosovo Serbs say the declaration of independence by Pristina is illegal.
Two days after the proclamation, Kosovo Serbs torched two border crossings with Serbia and have since staged a series of other protests.
Russia, which opposes Kosovo's independence move, said on Monday that the clashes stemmed from from Pristina's decision to secede from Serbia.
"The rise in tension there, above all in the ethnic Serb districts, is a direct consequence of the unilateral declaration of independence by Pristina and the non-acceptance of this illegitimate step by Serbs living in Kosovo," a statement by the country's foreign ministry said.