It's fascinating to understand the origin of political movements.
The cool thing about political parties in the 1800's was that they tended to be conglomerations of different movements rather than the well rehearsed national talking points we're used to. The Whig party that Fillmore belonged to was originally made up of several smaller, more focused interest groups.
The Whigs pulled together supporters in the Northeast that were fearful of immigrants and catholics (the No Nothing Party), northerners and westerners opposed to slavery because of various moral and economic reasons (the Free Soil Party) and anti aristocratic movements (the Anti Mason movement).
The other groups that eventually merged into the Whig party were manufacturing interests that were supportive of tariffs on imports and citizens that felt the county needed to be governed in a more modern matter that didn't match with Jefferson's original vision of an agrarian, egalitarian society which the Democrats still held. In the South, they tended to attract supporters who were skeptical of the expansion of Executive power under Jackson who wanted Congress to be the most powerful branch of government.
The one thing that united all Whigs was that they hated Andrew Jackson. For the entire life of the Whig party which was from around 1830 to 1865, 'King Andrew' remained their manifestation of what they should fear the most, even years after his death.
This is similar to the modern incarnation of Republicans rising up against FDR and his liberal, new deal policy. Powerful movements can be started from love or more often hatred of a powerful personality.