Relations of ideas are propositions like those of mathematics and logic: "3 + 4 = 7", "all bachelors are unmarried", "the square on the hypotenuse equals the sum of the squares on the other sides." Their truth is discoverable by reason alone, without any appeal to experience. To deny them is to contradict yourself. They are necessarily true — but they tell us nothing about the world.
Matters of fact are propositions about how things actually are: "the sun will rise tomorrow", "fire is hot", "Caesar crossed the Rubicon." Their truth is never self-evident and can always be imagined otherwise without contradiction. They depend on experience, not pure reason.
The fork's edge falls hardest on metaphysics. Claims about God, the soul, free will, and necessary being are neither relations of ideas (they are not provable by logic alone) nor matters of fact (no experience could verify or falsify them). They fall between the two prongs and come up empty. Hume's fork is one of philosophy's most powerful instruments of demolition — an empiricist test that much of traditional philosophy simply fails.
Hume's Fork is named by later philosophers; Hume himself simply calls the distinction one between "relations of ideas" and "matters of fact." Kant's reading of this passage famously woke him from his "dogmatic slumber" and led to the Critique of Pure Reason.