"One of the agents did not pay one of the prostitutes, and she complained to the police," Kessler told cable news channel CNN. "This is clearly the biggest scandal in secret service history," Kessler added.

"Because of this, those personnel are being relieved of their assignments, returned to their place of duty, and are being replaced by other secret service personnel. The secret service takes all allegations of misconduct seriously," the statement said.

The scandal is a major blow to the secret service, which prides itself on discretion and professionalism and has a reputation for taking no chances with presidential security.

The story is especially damaging as several bomb blasts rocked Cartagena on Friday during preparations for Obama's arrival at the Summit of the Americas. Though they occurred far from the summit, it was a reminder of the gang and drug problems the country faces and the need for heavy security in the city.

A heavy Colombian army presence has also been deployed, including thousands of troops and police and even submarines off the coast.