Scope

The ability to look beyond what is learned and apply the learned information to new scenarios often distinguishes animals from AI artifacts. An important component of this ability is recognizing novel combinations of previously learned patterns. Despite their flexibility and ability to be trained for virtually any specific task, AI algorithms tend to be limited to the training scenario. Applying one model to several scenarios may require training sets which become combinatorially impractical.

Ideal models should use minimal degrees of freedom and apply to maximal the number of scenarios or novel combinations (without extensive retraining for each). We will discuss how to encorage the design and the evaluation of methods to better apply learned information to new scenarios.

Goals of Workshop

1. Generate simple data sets revealing Human Level AI type problems.

The test set would be similar in spirit to Caltech 101 - a test set for computer vision - but for Human Level AI. The test set should be as simple as possible and is to be posted on the web as a benchmark.

2. How to evaluate generalizability of human level AI algorithms.

We will discuss criteria to estimate degrees of freedom in algorithms. The end goal is to be able estimate a ratio of scenarios applicable vs. degrees of freedom for algorithms proposed toward Human Level AI. Such metrics are intended to facilitate comparisons across AI methods.

Format

This workshop aims to provide a forum for researchers interested in improving the design of AI algorithms. By gathering in a friendly environment, the hope is that researchers can openly share their ideas and vision for the future of this field.

We will have two to three talks describing 1) the most basic tasks that show divergent human and AI performance. 2) Evaluating the robustness and degrees of freedom inherent in various methods. Between talks we will have discussions on how best to design simple test sets and 2) evaluate AI systems. We will collect perspectives from the audience along with the panelists. Using this feedback we will determine: test sets, definitions and strategies to evaluate algorithms.

Panelists

Links

Related Links:

• IJCNN 2009 Tutorial: Forms of Plasticity Motivated by Neuroscience