Passengers who resisted the smugglers were stabbed or beaten with wooden and steel clubs, then thrown overboard where some were attacked by sharks, the UN high commissioner for refugees said, citing survivors.
"Several recovered bodies showed signs of severe mutilation," UNHCR said.
"Survivors also reported that several Ethiopian women and at least one Somali were raped and abused by the smugglers during the voyage from Bosaso in Somalia's Puntland region.
Survivors also alleged that some Yemeni security forces confiscated their money once they reached shore.
Several boats involved
Yemeni officials said several boats were involved in Thursday's incident.
The dead and missing came from a single wooden boat that was carrying around 120 people, said an official from the local government in coastal Shabwa province.
Eight bodies were found on Thursday, and another 23 were taken out of water over the past couple of days, the official said.
Another 450 people forced out of other boats at the same time have been rescued, he said.
The UNHCR put the number of dead at 29, with 71 missing.
Smuggler brutality
It was the latest case of smuggler brutality involving boats carrying people across the Gulf of Aden from war-torn Somalia and the Horn of Africa.
It brings the total number of dead and missing among people trying to reach Yemen so far this year to 264.
"We are horrified by this latest tragedy," said Erika Feller, the UNHCR's assistant high commissioner.
The victims are people "who are desperate to escape persecution, violence and poverty in the Horn of Africa," she said.
A similar incident occurred on Saturday, when smugglers forced 140 people into the water elsewhere along the coast. All the migrants were believed to have been rescued, Yemeni officials said.