 for the expected image pair (row $2$). Since the chosen image pair is unknown *a priori* and the model is fully convolutional, the initial prediction is the constant gray background when the first image appears. The model then rapidly copies this image for the ensuing three frames. Next, the model successfully predicts the transition to the second object (a stack of tomatoes in this case). In row $3$, a sequence that differs from the training pair is presented. The model still makes the prediction of a transition to tomatoes, even though a chair is presented, but then copies the chair into subsequent predictions. Fig. \[image\_pairing\] shows that the unexpected transitions result in a larger average response in the final $E$ layer of the network. In fact, in all levels and all unit types ($E$, $A$, $R$), there is a larger response to the unpredicted images (Supp. Table 1). The overall magnitude of the difference is similar for $A$ and $E$, and is lower for $R$.

#### Norm-Based Coding of Faces

The representation of deviations from expectations can also explain observed neural embeddings of familiar stimuli such as faces. The norm-based coding theory suggests that faces are encoded with respect to a mean face [@Rhodes_2006]. Leopold et al. [@Leopold_2006] demonstrated that face-responsive neurons in macaque anterior IT are frequently tuned monotonically (often positively) to directions away from an average face (Fig. \[norm\_faces\]). This was tested by using synthetic faces with continuously varying levels of caricature. Training for next-frame prediction of rotating, computer-generated faces [@Lotter_2017], we see similar effects in the PredNet. The faces were created using software implementing a principal component analysis of a corpus of human faces (FaceGen [@facegen]). For training, $16K$ sequences were generated with a random face, initial orientation, and rotation velocity. The blue curve in Fig. \[norm\_faces\]b illustrates the post-training response of $E$ units in the network as a function of car