A dispute over an alleged robbery in Rio de Janeiro has escalated after two Team USA swimmers were pulled from their flight back to the United States.

U.S. Olympic Committee spokesman Patrick Sandusky confirmed Jack Conger and Gunnar Bentz were removed from the plane by Brazilian authorities, but shared no other details.

Swimmer Ryan Lochte made global headlines after he told reporters that he and three other team members were robbed at gunpoint in Rio early Sunday morning.

This Tweet is currently unavailable. It might be loading or has been removed.

But, now, Lochte is changing his story.

The foursome were held up by armed men dressed as police as they returned home in a taxi after a night of partying Saturday, Lochte originally told Today in an interview. They took the group's money and Lochte's wallet, holding a gun to his forehead, but left his cellphone and ID.

In a story that aired on NBC Wednesday night, the network reported that there were discrepancies between the swimmers' stories, and that images shared on social media contradicted their original timeline of events. NBC also aired surveillance footage that Brazilian authorities say shows the American swimmers returning to their lodgings in good spirits, showing no signs of distress.

Then, in a later interview, Matt Lauer said that Lochte changed a few details from his story, now saying the gun was pointed in his direction, not pressed against his forehead.

Lauer said he asked Lochte if he had made up a story to cover up certain behavior and, Lauer says, the swimmer "strongly denied" that allegation. Lochte added the swimmers didn't originally want to report the incident out of fear that they had broken USOC rules by being out drinking.

This Tweet is currently unavailable. It might be loading or has been removed.

This Tweet is currently unavailable. It might be loading or has been removed.

Read a breakdown of other discrepancies here.

Brazilian authorities are skeptical of Lochte's original allegations, and have so far been unable to track down the taxi driver involved or find further evidence to corroborate the swimmers' accounts.

The International Olympic Committee has also denied the incident occurred.

In order to get to the bottom of the incident, Brazilian Judge Keyla Blank on Wednesday ordered Lochte and James Feigen, another swimmer involved in the incident, to stay in the country and for their passports to be confiscated.

Lochte had already left the country to return to the United States, according to his father Steve Lochte. Feigen remains in Brazil, the Brazilian publication Globoreported, and will be summoned to answer questions.

If Feigen remains, he'll have some company in Conger and Bentz, who are being forced for the moment to remain in Rio.

This Tweet is currently unavailable. It might be loading or has been removed.

This Tweet is currently unavailable. It might be loading or has been removed.

A video posted by Globoshowed Conger and Bentz declining to answer questions from a reporter as they entered a police station.

Brazil has been under the microscope regarding the safety of athletes and visitors during the Games. Authorities are perhaps understandably concerned that Lochte's story, if it turns out to have been made up or exaggerated, has unfairly impugned the city's reputation and sullied the success of the country's Olympics.

The Associated Press contributed reporting.