Millard Fillmore was a decent president that just happened to take over after a sitting president died in office and didn't really have any popular mandate for enforcing his policy. The nation was in that tense period between the Spanish American War and the Civil War and needed a steady hand to guide the ship of state.
While Fillmore didn't do anything extraordinary or horrible during his presidency, Pierce seemed to do everything he could to stir resentment in the North and destroy the fragile peace that existed between the North and the South.
Pierce's campaign slogan was "We POLKed you in 1844, we shall Pierce you in 1853" and he seemed to do just that.
Though he was from New Hampshire, Pierce consistently sided with the concerns of the slave holding aristocracy in the South. While Fillmore can be faulted for avoiding the explosive issue of slavery, Pierce put it in every one's faces, enforcing the Fugitive Slave act in Boston with Federal Marshalls. He worked hard to make sure that Haiti did not receive official diplomatic recognition by the United States to avoid the impression that it was acceptable for slaves to revolt. He refused to send Federal troops in to the territory of Kansas to help the abolitionists that settled there from being targeted by Southern death squads.
All of these things continued to embolden the South and embitter the North, making it more clear to each side that their real enemies were now internal instead of the external enemies they faced up until the 1840's. Pierce goes to show that sometimes it's better for a President to do nothing than something.