The words “Whig” and “Tory” refer, in J.S. Mill’s time, to the two primary political parties of England. Whigs comprised the liberal party while the Tories were (and still are) conservative. The Whigs first emerged in 1679 as the Country party, a small group of reformers born of the Cavalier Parliament that Charles II instated in 1671 and dismissed eight years later. The Country party opposed the corruption of Parliament and its susceptibility to bribery. In 1679, the group, by then the Whigs, began to battle the church’s persecution of Protestant Nonconformists. In response, the Tory party, which voiced its defense of the church and of the monarchy, was formed in 1681. This legacy of opposition was maintained in the early 19th century when the two groups continued to disagree on economic and social issues. Whigs supported free trade and parliamentary reform among other causes; the Tories stood against their objectives.
Mill considered himself to be more liberal than the Whigs. He called himself “as much as ever a radical and democrat, for Europe, and especially for England,” which was relatively conservative in both political parties. Some of Mill’s largest critiques of this conservatism, particularly Tory conservatism, surrounded its fixation on hereditary power through monetary means and its disavowal of women’s suffrage. He said that he “thought the predominance of the aristocratic classes, the noble and the rich, in the English Constitution, an evil worth any struggle to get rid of…” thereby discussing the manner in which conservatism and the common law that protected it upheld the power of the wealthy. About women’s suffrage, Mill said that “every reason which exists for giving the suffrage to anybody demands that it should not be withheld from women,” a notion too liberal even for Mill’s radical father.
Sources
https://www.nationaltrust.org.uk/features/who-were-the-whigs
https://www.britannica.com/topic/Whig-Party-England
Penguin Edition of Autobiography by J.S. Mill: Own summary as well as quotes drawn from pages 136 and 93.