Mahir Ahmad Mahmud al-Zubaydi, also known as Abu Assad or Abu Rami, was killed along with an unidentified woman in Baghdad's Sunni district of Adhamiyah on Friday, a statement said.
Intelligence reports led US forces to a building in Adhamiyah where Abu Rami was holed up.
The troops came under attack from inside the building and fired back, killing Abu Rami and the woman.
The troops later found explosives inside the building and detonated them, according to the statement.
The killing of Abu Rami has dealt a severe blow to Al-Qaeda in Baghdad, US spokesman Admiral Patrick Driscoll said.
"His removal from the AQI (Al-Qaeda in Iraq) network will send shockwaves through Baghdad's terrorist bombing networks," he said in the statement.
"Its ability to conduct grisly attacks against Iraqi civilians and coalition and Iraqi forces has been severely crippled by this precision operation."
The military statement said Abu Rami's group was responsible for suicide bomb attacks in Baghdad on Thursday.
In one incident, a man blew himself up at the gate of Al-Rasool mosque in the Jadida neighbourhood and killed 12 people.
A suicide car bomb on the same day in the Zafaraniyah neighbourhood killed eight people, including three Iraqi soldiers.
Abu Rami was reportedly also responsible for multiple car bombings and mortar attacks in Sadr City in 2006 and 2007, including the car bombings on November 23, 2006 that killed more than 200 people.
The latter attack, in the sprawling Shiite district of some two million people, was one of the deadliest assaults in Baghdad during the peak of sectarian violence across the capital.
Abu Rami was also alleged to have planned and participated in several kidnappings and executions.
"In a video recording from June 2006, Abu Rami is seen shooting one of four Russian diplomats," the statement said.
On June 3, 2006, a group of Russian diplomats were attacked by gunmen in the the upmarket Baghdad district of Mansur.
One diplomat Vitaly Titov was killed in the attack and four others were kidnapped and later killed.
An insurgent alliance led by Al-Qaeda later claimed it had carried out the murders of the Russians after Moscow failed to heed an ultimatum to pull out of Chechnya and free Muslim prisoners from Russian jails.
Abu Rami had ties to senior leaders of Al-Qaeda in Baghdad and was also responsible for attacks in Baghdad, Salaheddin and Diyala provinces, the statement said.
Originally a member of Ansar al-Islam, Abu Rami joined Al-Qaeda in 2004 and became the group's head of Baghdad's Rusafa district in 2007.
The military said Abu Rami specialised in car bombings and suicide bombings and personally selected the targets.
"He allegedly ordered and directed a car-bombing attack May 1 that killed one American soldier and wounded three soldiers and three Iraqi civilians. Detainee reporting puts him at the scene of the crime," the military said.
In a separate statement the military said its troops have detained 10 other suspected Al-Qaeda militants in raids across northern and central Iraq since Friday.