Mill is not an anarchist. He grants that society has real claims on its members — the obligation to defend it from injury and to bear a fair share of its burdens. Where a person's conduct damages others or fails duties owed to others, society may intervene, through law if necessary. The question is not whether society may ever restrict freedom but what counts as a genuine harm to others, and Mill is consistently more restrictive than his contemporaries in answering this question.
The most important category in Mill's analysis is self-regarding conduct — action that affects primarily the person acting. Society has no right to coerce here, even if the conduct is unwise, imprudent, or morally objectionable in the eyes of the majority. The fact that others disapprove is not a harm. The fact that others worry is not a harm. Harm, in Mill's sense, requires a specific, assignable injury to interests that deserve legal protection.
Mill adds that even when society cannot coerce, it may still advise, persuade, argue, and publicly reason with individuals about their choices. The harm principle does not insulate people from criticism or judgment — it only insulates them from compulsion. The opinions of others are a legitimate influence; the power of others is not.
Mill's strongest practical argument against social interference in personal life is not principled but empirical: when society interferes in self-regarding conduct, it almost always interferes wrongly. On questions about what is good for individuals in their own lives, the majority has no special competence. Each person knows their own circumstances, temperament, and values better than any external observer. Social opinion on personal matters is characteristically the product of prejudice, custom, and the discomfort of the majority with whatever is unfamiliar — not of genuine wisdom about what is good for the person in question.
Chapter IV of On Liberty sets out the principle that society may only compel those actions which concern others, and addresses the most common objections. Mill's five chapters move from theory to practice, with Chapter V applying the principles to specific cases such as trade and the sale of intoxicants.