Contact Roeland Ordelman
Organizations University of Twente


Instead of typing keywords in a search bar, we would rather have a chat with a (virtual) librarian and express our ‘information need’ or interest in certain content in the archive in a conversation. At Netherlands Institute for Sound and Vision, we investigate several use cases in the area of Spoken Conversational Search (SCS) as a means to guide access to our rich but also large and heterogeneous audiovisual archive. In our LABS environment, we have search APIs built on top of indices of rich metadata that can be connected to a dialogue system that uses keyboard or speech input to communicate with users online and in our museum.

Thanks to progress in AI, natural language dialogues between humans and conversational agents are becoming a realistic option to help users of an archive such as NISV to express their information need. Especially in complex search tasks that require users to reference or be aware of the attributes of a collection (metadata), SCS dialogues can be used to guide users filling those attributes (facets) in the course of a mixed-initiative natural conversation.

Research questions we can think of:

  • how to represent the ‘librarian’s knowledge’ of the contents of the archive so that it can be used in an SCS system?
  • how to model initial utility to users of collection items?
  • which type of use does benefit the most from an SCS approach?
  • What are the ethical and legal aspects of conversational AI?
  • how to encode multimodal dialogue status and model the evolving ‘information need’ of a user that communicates with the system and receives information from it?
  • would it already be possible to develop a ‘convincing system’ for museum visitors with a simple approach that uses a search API and its metadata (e.g., facets from the result list)?
  • how to personalise a dialogue based on additional information sources (e.g., a camera that spots a child)?
  • how to improve the natural language processing of such a dialogue system (e.g., in a Museum environment)?
  • how to implement a search dialogue in a VR environment with the librarian as an omnipresent oracle?
  • could we enrich the information of archival content by using the information we obtain from the human-computer dialogue?

Please contact us if you are interested to engage with our data and APIs for your research or development project.