 can obtain the most information gain. A question [$q\!=\!(I,\hat{v},\Lambda_{\hat{v}})$]{} requires people to determine whether our approach predicts the correct part template $\hat{v}$ and parses a correct region $\Lambda_{top}=\Lambda_{\hat{v}}$ for the part. Our method expects one of the following answers.

**Answer 1:** the part detection is correct. **Answer 2:** the current AOG predicts the correct part template in the parse graph, but it does not accurately localize the part. **Answer 3:** neither the part template nor the part location is correctly estimated. **Answer 4:** the part belongs to a new part template. **Answer 5:** the target part does not appear in the image. In particular, in case of receiving Answers 2–4, our method will ask people to annotate the target part. In case of getting Answer 3, our method will require people to specify its part template and whether the object is flipped. Our method uses new part annotations to refine (for Answers 2–3) or create (for Answer 4) an AOG branch of the annotated part template based on Equation (\[eqn:LossAOG\]).

### Question ranking

The core of the QA-based learning is to select a sequence of questions that reduce the uncertainty of part localization the most. Therefore, in this section, we design a loss function to measure the incompatibility between the AOG and real part appearances in object samples. Our approach predicts the potential gain (decrease of the loss) of asking about each object. Objects with large gains usually correspond to not well explained CNN neural activations. Note that annotating a part in an object may also help localize parts on other objects, thereby leading to a large gain. Thus, we use a greedy strategy to select a sequence of questions [$\Omega=\{q_{i}|i=1,2,\ldots\}$]{}, *i.e.* asking about the object that produces the most gain in each step.

For each object image $I$, we use [${\bf P}(y|I)$]{} and [${\bf Q}(y|I)$]{} to denote the prior distribution and the estimated distribution of an object part on $I$, respec