We propose a challenge task to identify the correct food item(s) from a semantically annotated food knowledge graph given a question. The questions can be of three types: (1) Factoid questions (e.g., "How much carbs are in a serving of breakfast burrito?"); (2) Comparison questions (e.g., "Does Peanut Oil have more monounsaturated fat compared to Sesame Oil?''); and (3) Constraint questions (e.g., "Suggest a food item without peanuts?"). The challenge participants will have access to various semantic knowledge sources on food and an evaluation dataset for the above types of questions with sample answers. The range of questions used in the challenge will consist of eliciting food concepts to understand the relations between food, their ingredients, nutritional values, human health, and any contextual information applicable to a food recommendation.
There is an immense amount of food-related information on the Web, including structured semantic data sources such as DBpedia and Wikidata. Recent efforts have brought together even more semantic data sources on food, such as the FoodOn, Healthy Lifestyle Support Ontology (HeLiS) ontology, and the FoodKG. There are some other structured data sources, such as the FoodBase, which is annotated with the Hansard corpus, and the FoodOntoMap that has a normalized set of semantic tags from different food ontologies. In addition, the BuTTER method is a bidirectional Long Short-Term Memory (LSTM) with Conditional Random Field (CRF) that can be utilized for food named-entity recognition. Even though there is an increase in such semantically annotated rich datasets on food, we have not yet seen wide-scale unified adoption of these resources, such as in question answering, that this challenge will address. Given the increase in applications of question answering systems in the recent past, we believe there will be much enthusiasm from the ISWC community for this challenge.
Food is a universal aspect of our lives. Answering questions about food that are semantically annotated can easily showcase advances in semantic web and information retrieval techniques. The challenge participants will be expected to answer three types of questions related to food using the challenge datasets. The question types include the following.
Factoid questions that can be easily answered by a direct lookup (e.g., "How much carbs are in a serving of breakfast burrito?").
Comparison questions that require evaluation of two food concept entities with respect to one or more of their attributes (e.g., "Does Peanut Oil have more monounsaturated fat compared to Sesame Oil?").
Constraint questions that will have one or more restrictions on the food type that should be returned (e.g., "Suggest a food item without peanuts?"). The constraints may include one or more of the following: (I) User attributes (e.g., "peanut allergy" that will require an advanced inferring to ascertain what specific foods to avoid); (II) Substitutions that require determining a comparable food item through explicit (e.g., owl:sameAs, rdfs:seeAlso relationships) or implicit semantic relationships (e.g., inferring similar foods based on the attributes of a recipe or its ingredients); (III) Other contexts (e.g., food pairing such as a wine to go with spicy food) ; (IV) User preference (e.g., like crispy food, can only afford foods under a certain cost); (V) External factors such as seasonal ingredient availability (e.g., foods abundant during the summer). Some of these constraints can be thought of as hard constraints (e.g., "peanut allergy"), while others can be considered as soft constraints (e.g., user preference to "crispy texture" of a food.)
The questions presented to the challenge participants will be in natural language. The questions may follow several pre-specified templatized patterns, especially for the factoid and comparison questions, should the challenge participants wish to utilize SPARQL queries to answer the questions. However, the conversion from natural language to the SPARQL query is optional. We will accept solutions that utilize natural language processing techniques that directly use the question, as long as the answers are obtained from the food knowledge graph provided. All the entity and attribute names included in the question will be available in this graph, though many of the relationships may need to be inferred. The complex constraint questions will require some context information and reasoning, and the constraints in such questions will be given as several long lists of non-conflicting constraints.
The challenge organizers are already maintaining several large-scale food-related knowledge graphs, including the FoodKG, HeLiS, the FoodBase, and the FoodOntoMap. Collectively these datasets contain close to 2 million concepts, and they will be used in the challenge. Furthermore, a dataset containing food-related question and answer pairs will be released to the challenge participants to evaluate their work. This dataset will be bootstrapped from question-answer pairs pertaining to food related information from the RecipeQA, Grocery and Gourmet Food, QALD-9, LC-QuAD v1.0, and LC-QuAD v2.0 datasets. Furthermore, the challenge dataset will be enriched with question-answer pairs collected from publicly available websites that focus on evidence-based dietary recommendations, food composition, food consumption data, and food-health relations. We estimate to have around 10,000 food-specific Q&A pairs in the dataset that will be provided to the challenge participants.
The challenge participants are expected to provide the answer(s) to each natural language question given. In the case there are multiple answers, the challenge participants must rank them in the best possible order. We will only consider the top $k$ ranked list of answers provided, where k is defined per question supplied in the test dataset. All returned recipes must satisfy the question criteria, especially all the \emph{hard} constraints, but may relax one or more of the soft constraints. Computation time must be reported and should not exceed $t$ seconds as defined per question in the test dataset. The hardware used to train and test the models must also be specified. The answers will be evaluated using the Mean Reciprocal Rank (MRR) and the Macro F1 measure. The challenge submissions will be ranked based on the F1 score that will be automatically evaluated using an evaluation script that we will provide and made available on our public GitHub challenge repo. We plan to expose the challenge evaluation script via a REST API, and create a website with a leaderboard. Furthermore, to make the challenge presentations more enjoyable, we plan to have esteemed senior semantic web, and food researchers try out the top challenge submissions and give their feedback at the challenge session at the ISWC'21 conference.
To be announced . . .
Rensselaer Polytechnic Institute, USA
IBM Research, USA
IBM Research, USA
Fondazione Bruno Kessler, Italy
Jožef Stefan Institute, Slovenia