A goal constraint treats something as a good that should be promoted: maximise the total respect for rights in the world, minimise rights violations overall. A side constraint treats something as an absolute limit: do not violate this person's rights, full stop — regardless of whether doing so would produce better outcomes. Nozick insists that rights are side constraints. You may not violate one person's rights to prevent two others from having their rights violated, because each individual is the kind of being whose separateness must be respected absolutely, not instrumentalised for aggregate benefit.
The philosophical foundation of rights as side constraints is what Nozick calls the separateness of persons — the Kantian insight that each person is an end in herself, not merely a location of welfare or a vessel for the good. Utilitarian moral theories that aggregate welfare across persons treat the population as one large person whose total well-being is to be maximised; this ignores the fact that there is no super-person who experiences the aggregate. The person sacrificed for the greater good does not get compensated by the gains to others; those gains go to different people. Rights as side constraints take this separateness seriously as a moral fundamental.
From the side-constraint view of rights, Nozick derives the minimal state. Any state that taxes individuals to provide benefits they have not chosen — redistributive taxation, welfare programmes, compulsory education — violates the side constraints by using individuals as means to collective ends. Even if the outcomes are good by most measures, the process violates rights that may not be violated. The only legitimate state is one that restricts itself to protecting the rights it is in business to protect: protection against force, theft, fraud, and enforcement of voluntary contracts. Everything else — however benevolent in intent — constitutes rights violation.
The side-constraint framework is introduced in chapter 3 of Anarchy, State, and Utopia, where Nozick explicitly contrasts it with utilitarian goal constraints. The concept draws on Kant's formula of humanity but gives it a more libertarian application than Kant himself drew.