Standing not far from the Berlin Wall that once divided the city, Obama said "the walls between old allies on either side of the Atlantic cannot stand".
He also urged Europeans and Americans to work together to "defeat terror and dry up the well of extremism that supports it".
"Now is the time to build new bridges across the globe as strong as the one that bound us across the Atlantic."
"The walls between races and tribes; natives and immigrants; Christian and Muslim and Jew cannot stand. These now are the walls we must tear down," he said.
Ending Iraq war
The Illinois senator was speaking at the start of the European leg of his international tour, which has also taken him to Afghanistan, Iraq, Jordan, Israel and the West Bank in a bid to bolster his foreign policy credentials ahead of the November US presidential election.
Police in Berlin said Obama's speech drew more than 200,000 people from across Germany.
The Illinois senator also said the US and Europe must stand together in telling Iran to abandon its nuclear ambitions and insisted "we must renew our resolve'' to defeat the Taliban in Afghanistan.
Obama drew applause from the crowd when he said that the war in Iraq must end, although he did not give any specific timeline.
"This is the moment when the world should support the millions of Iraqis who seek to rebuild their lives, even as we pass responsibility to the Iraqi government and finally bring this war to a close," he said.
McCain critcism
Earlier, Obama held talks with Angela Merkel, the German chancellor, followed by meetings with Frank-Walter Steinmeier, the German foreign minister, and Klaus Wowereit, the Berlin mayor.
He will go on to France and Britain after concluding his Germany visit.
John McCain, the US Republican presidential candidate, dismissed Obama's speech on Thursday as irrelevant to US voters.
"I'd love to give a speech in Germany. But I'd much prefer to do it as president of the United States rather than as a candidate for president," McCain said.
No 'political rally'
On the flight to Berlin from Israel, where he just completed his Middle East tour, Obama said he wanted the US and Europe to rediscover their common ground.
"There is no doubt that part of what I want to communicate on both sides of the Atlantic is the enormous potential of us restoring a sense of coming together," he said.
He played down comparions with the late president John F Kennedy's "Ich bin ein Berliner" vow to the then-divided city in 1963, or Ronald Reagan's call in 1987 to "Tear down this wall".
"They were presidents, I am a citizen," he said.
But Obama defended himself against claims that he is defying convention by campaigning abroad, saying he wanted to speak to the whole of Europe so he needed a big venue.
"The people in the crowd are not voters, in that sense, it is not designed to get them to the polls," he said.
"It is not a political rally, hopefully it will be viewed as a substantive articulation of the relationship I would like to see between the United States and Europe."