tations by computing adversarial samples for a CNN. In particular, influence functions [@CNNInfluence] were proposed to compute adversarial samples, provide plausible ways to create training samples to attack the learning of CNNs, fix the training set, and further debug representations of a CNN. [@banditUnknown] discovered knowledge blind spots (unknown patterns) of a pre-trained CNN in a weakly-supervised manner.

Zhang *et al.* [@CNNBias] developed a method to examine representations of conv-layers and automatically discover potential, biased representations of a CNN due to the dataset bias. Furthermore, [@wu2007compositional; @yang2009evaluating; @wu2011numerical] mined the local, bottom-up, and top-down information components in a model for prediction.

**CNN semanticization:**[` `]{} Compared to the diagnosis of CNN representations, semanticization of CNN representations is closer to the spirit of building interpretable representations.

Hu *et al.* [@LogicRuleNetwork] designed logic rules for network outputs, and used these rules to regularize neural networks and learn meaningful representations. However, this study has not obtained semantic representations in intermediate layers. Some studies extracted neural units with certain semantics from CNNs for different applications. Given feature maps of conv-layers, Zhou *et al.* [@CNNSemanticDeep; @CNNSemanticDeep2] extracted scene semantics. Simon *et al.* mined objects from feature maps of conv-layers [@ObjectDiscoveryCNN_2], and learned explicit object parts [@CNNSemanticPart].

Unlike above research, we aim to explore the entire semantic hierarchy hidden inside conv-layers of a CNN. Because the AOG structure [@MumfordAOG; @MiningAOG] is suitable for representing the semantic hierarchy of objects, our method uses an AOG to represent the CNN. In our study, we use semantic-level QA to incrementally mine object parts from the CNN and grow the AOG. Such a “white-box” representation of the CNN also guided further active QA. With clear semantic struc