At least one person was injured when the multi-storey building containing thousands of historical records collapsed on Tuesday.
"According to witness reports, we have to assume that there are still two to four people missing and possibly still underneath the rubble," Gregor Timmer, a spokesman for the city of Cologne, said.
"There are reports on two inhabitants of the building next to the archive that collapsed and two people who were inside a car in front of the building at the time of the tragedy."
The cause of the collapse was unclear.
Local media reports have pointed to construction on an underground train line.
But a spokesman for the city's metro line in the area denied suggestions that work on the line could have caused the collapse.
"There was no work taking place directly underneath the building," he said on Germany's N-TV.
"As far as I know there was no building work taking place that could have
caused this to happen."
Strange noises
The historical archive was one of the biggest of its kind in Germany and housed, among other, papers from Heinrich Boell, the Nobel Prize winning German author.
Witnesses said the building appeared to groan and creak before collapsing.
"I heard a giant bang and then suddenly saw this giant gray dust cloud. It was like a Hollywood film," Mustafa Goresme, who watched the archive fall as he sat in a restaurant, said.
Susanne Vandenberg was drinking a coffee inside the archives when it began to collapse.
"We all ran out of the archives. I turned round and the front of the building started to collapse, the first windows started falling down. So I started to run. There was a huge cloud of dust behind me," she told Germany's WDR television.
Florian Hacke, who lives two buildings away from the archive, said he ran out of his house after he heard a creaking noise and cracks opened up in his ceiling.
Eberhard Illner, former director of the archives, said cracks had appeared in the building in recent weeks.
Firefighters at the scene said it appeared that a large underground room had been dug close to the building, and television reports showed a huge crater nearby.