Three bodies were found north of Cape Breton, Nova Scotia, later on Saturday.
Two other crew members were rescued by a nearby fishing vessel.
A spokesman for the Canadian coast guard told reporters: "Three deceased members of the crew were recovered from the water."
The boat was carrying seal hunters from the Magdalen islands before the accident.
Accident probe
The boat flipped over while it was being towed back to port by the coast guard after encountering steering problems.
The boat was on its way to the seal herds when it lost a rudder, prompting an SOS appeal to the coastguard.
Canada's transport ministry will launch an investigation.
The ministry said on Saturday that the Acadien had passed inspection at the beginning of the week.
Canada's annual harvest of harp seals started on Friday, with several boats setting off from the tiny town for the seal herds.
Animal groups
Michel Plamondon, a fisheries department spokesman, told AFP news agency: "Yesterday, there were 16 boats [that set off on the sea hunt] but today, because of the accident, most of them have decided to return to the Magdalen islands."
Only three or four boats remained at sea to pursue the hunt, with weather conditions expected to remain difficult for the next two days, he said.
Hunters routinely face shifting ice, high winds, freezing temperatures and unpredictable seas during the controversial sea hunt, which is often protested by animal rights groups.
Canada set the limit for this year's harvest at 275,000 harp seals, 5,000 more than the previous year.
About 800 seals were killed on Friday Plamondon said.