He was responding to a report in the New York Times that revealed Justify had tested positive for the substance scopolamine after winning the Santa Anita Derby on April 7.
The New York Times alleges that if California officials had followed the rules the failed test would have disqualified the colt from running in the Kentucky Derby, but a delay in the process enabled him to compete.
He later become only the second horse in four decades to land the Triple Crown.
"It's a damn shame it happened, especially to that horse," Baffert said.
"I've never administered that drug or had it administered to one of my horses. I wouldn't even know what form it would come in.
"It's not fair to the horse, the connections, or to me. I don't feel it diminishes his accomplishments because he got tested before that and after that.
"We're always getting notices to be aware of that stuff, but looking for it is like looking for a needle in a haystack.
"Fortunately I've never had one of these before, but I've seen it happen to other trainers.
"You hold your breath that it doesn't happen to you, but we're sitting ducks. Contamination is hard to control.
"It's common sense that nobody would intentionally give their horses something like scopolamine. I wouldn't do that."
It reportedly took the California Horse Racing Board more than a month to confirm the results. It then opted to drop the case.
Rick Arthur, an equine medical director for the California Horse Racing Board, has defended their actions.
"Even a high Bute case can take 60-90 days to resolve. There's no way this case could have been resolved prior to the Kentucky Derby," he said.
Arthur said Scopolamine is found in jimson weed and is common in California.
"This is not a case of someone drugging a horse. This is a case of a horse poisoning. The source of it is a poisonous plant.
"The board, on my and the executive director's recommendation, made the correct, appropriate and the gutsy decision to dismiss the case."
Part-owner Elliott Walden, president of Winstar Farm, said his operation was informed of the failed post-race drug test in mid-April and turned the matter over to its attorneys.
He said they never heard anything further from the regulator.
"It's ridiculous it's even being brought up now," Walden said.
"Scopolamine is a known contaminant in California. Other top trainers have dealt with this.
"It's a shame for Bob's reputation, Justify's reputation, and our reputation."
Churchill Downs and the Kentucky Horse Racing Commission also say they had no knowledge of any potential positive tests that may have emanated from California in advance of the 2018 Kentucky Derby.
All pre- and post-race tests for 2018 Kentucky Derby participants came back clean including Justify.