Zero-shot guessing game

Assuming that the model has learned to compose concepts during the turns of the dialogue, we hypothesise that it should also be able to use these representations to play games involving target objects that belong to categories that have never been seen before. For example, humans can discriminate between a dolphin and a dog even though they might not know what it is called. The measure presented in this section has the potential to demonstrate whether current models lack the ability to systematically generalise to new instances that are composed of attributes learned during training.

We propose the Zero-shot gameplay setup to assess the ability of the model to generalise to novel reference scenes and target objects. In our paper, we show how, as show in Figure 1, current state-of-the-art model are unable to deal with unseen objects during training.

Please refer to our paper for more details about the reference games generation procedure. You can find the reference games used for the paper at the following Dropbox link.