Georgia accused Russia on Monday of a "campaign of harassment and persecution" in South Ossetia and Abkhazia, and urged the International Court of Justice to intervene to halt killings and forced expulsions.
Russia called the case "nonsense" and said the court has no jurisdiction.
The case opened a new legal front in the battle between Georgia and Russia for control of the regions, and it began as French President Nicolas Sarkozy led an EU delegation to Moscow for talks aimed at easing the standoff.
Georgia accuses Russian forces, local militias and mercenaries of conducting a campaign of murder, forced displacement and attacks on towns and villages that started in the early 1990s and culminated in last month's brief war.
Ethnic Georgians "are being forced out of their homes by a campaign of harassment and persecution," Tina Burjaliani, Georgia's first deputy minister of justice, told the court.
Georgia claims the campaign has left thousands of civilians dead and forced more than 300,000 from their homes.
Burjaliani said Tbilisi had filed its case "at a time of great distress in its history. A time when hundreds of thousands of its nationals are persecuted and displaced from their homes only because they are Georgians."
Burjaliani accused Russia of trying to undermine Georgia's independence "through a policy of divide and conquer ... that has ripped apart its delicate multiethnic culture."
Russia also accuses Georgia of crimes against humanity, for launching a massive attack last month on South Ossetia, killing Russian peacekeepers and dozens of civilians. Moscow says its military actions since are aimed at protecting its civilians. Before the war, Russia had given passports to many of the residents of South Ossetia, even though it is part of Georgia.
Outside the courtroom, Russia's ambassador to the Netherlands, Kirill Gevorgian, dismissed Georgia's case as "nonsense."
"The whole problem in Ossetia, in Abkhazia, is discrimination of Ossetians and Abkhazians by Georgia," he told reporters. "The line of Russia is to try to help the situation to keep the peace, to prevent the discrimination."
Moscow's lead lawyer, Roman Kolodkin, expressed regret for all victims of the conflict. But he told the judges the court "manifestly has no jurisdiction" and said Georgia had produced no evidence to support its claim of ethnic cleansing.
Russian leaders have bristled at the West for failing to condemn what they described as a Georgian "aggression" and indiscriminate killing of civilians, and threatened to prosecute Georgian President Mikhail Saakashvili as a war criminal.
A month after the outbreak of war in the region and weeks after a cease-fire was approved, Russian troops remain entrenched deep inside Georgian territory.
The dispute has plunged relations between Moscow and the West to near Cold War levels of animosity.
The 15-judge tribunal, unofficially known as the World Court, will likely take years to deal with Georgia's case, which accuses Russia of breaching the Convention on the Elimination of All Forms of Racial Discrimination.
"This is an extreme form of racial discrimination," said James Crawford, a lawyer for Georgia. "There has been burning of houses, murder of civilians, looting of property and forced expulsions on a scale that surpasses the darkest moments of the civil war of 1991-92," he added, saying 10 percent of Georgia's population has been displaced by the Russian campaign.
After three days of hearings that began Monday in the wood-paneled Great Hall of Justice in the court's seat in The Hague, judges must decide whether they have jurisdiction before mulling whether to impose any immediate measures to safeguard civilians. Even if they do, it is unclear whether Russia will comply, and the court has no way of enforcing its decisions.
Kolodkin said Russia and Georgia had no dispute under the convention and therefore the court lacked jurisdiction.
"The Russian Federation requests this court to declare it has no jurisdiction (and) to reject the request for provisional measures," he said.
Russia has recognized both South Ossetia and Abkhazia as independent, a move denounced in Georgia and abroad. The regions make up roughly 20 percent of Georgia's territory - and include miles (kilometers) of prime coastline along the Black Sea.
Payam Akhavan, another lawyer for Georgia, urged the court to call Russia into line.
"Violent discrimination has continued since the so called cease-fire, since Georgia filed its application and since the request for provisional measures was put before the court. It continues today," he said. "This court has an important role to play to help ensure it does not continue tomorrow."
More than 3,500 people were detained during unprecedented mass protests that swept across all of Russia in support of Alexey Navalny on January 23