Expressions can be ordered onto a circumplex revealing differing degrees of similarity between distinct expressions. We examine whether facial appearance (i.e., structural similarity and dissimilarity) predicts relationships between emotions. In Experiment 1, which examined ratings of emotions in faces, participants' ratings led to reliable similarity and contrast judgments between expressions. Experiment 2 employed a computational appearance model, which captured the visual statistics sufficient for synthesizing photorealistic faces in a set of variance dimensions. In Experiment 3, which indexed the expression validity of the appearance model, participants characterized synthetic expressions similarly to faces in Experiment 1. Finally, Experiment 4 examined humans' sensitivity to the top five dimensions, finding that certain dimensions predict subjective emotional relationships. These results reveal that capturing the signature of a specific emotion relies on its code across a set of underlying appearance dimensions, which may represent the fundamental anatomical features used to generate and recognize facial emotion.