Kevin Millwood (0-1) took a three-hitter and a 2-1 lead into the eighth. The right-hander retired the first two batters before Tejada — a career shortstop now playing third base — misplayed a sharp grounder by John McDonald.
Bautista then hit a 1-2 pitch into the left-field seats to put Toronto ahead for the first time. Gonzalez followed with his second homer of the game, a drive inside the foul pole in left.
That ended Millwood's second start since coming to the Orioles in an offseason trade with Texas. Before Tejada's error, the only runners to reach were Gonzalez, who homered leading off the fourth, and McDonald, who singled twice and finished with three hits.
Edwin Encarnacion made it 5-2 with a ninth-inning shot off Cla Meredith.
Casey Janssen (3-0) pitched the seventh after Toronto starter Shaun Marcum gave up two runs and eight hits in six innings.
Kevin Gregg finished the Blue Jays' first sweep in Baltimore since August 2004 by getting three outs for his second save.
Baltimore scored all its runs in the first inning. Tejada hit an RBI double and Luke Scott followed with a run-scoring grounder.
After wasting a leadoff triple by Adam Jones in the third, the Orioles got only one runner past first base the rest of the way.
Notes
Toronto has won five straight. … Baltimore reliever Koji Uehara (hamstring) threw 25 pitches off the mound in Sarasota, Fla., as part of his effort to return from the 15-day disabled list. … Gonzalez has four homers and at least one hit in all six games this season (9 for 26). … Orioles 2B Brian Roberts (abdominal strain) missed a second straight game and soon could be headed to the DL.