CS 598ea: Decision Making Under Uncertainty

The course covers decision-making algorithms and systems in situations of partial knowledge. The focus is on decision making in dynamic situations when the state of the world is not completely observable, and that state has to be estimated using partial knowledge of the way the world evolves (in response to actions or otherwise) and the way world states and actions give rise to observations. Specifically, we will cover exact and approximate decision making using the following formal systems and guiding problems:

Basic methods: Fully observed systems: Markov Decision Problems (MDPs), Planning, Reinforcement learning

Basic methods: Unobserved systems: Unobserved MDPs, Conformant planning, Nondeterministic planning

FOCUS OF CLASS: Partial observability and sensing actions: Partially Observable MDPs (POMDPs), Planning with sensing, Reinforcement learning in POMDPs

Branch-out topic: Knowledge in logic (modalities) and probabilities (distributions over distributions)

Branch-out topic: Adversarial situations: Markov games, Games of partial information

Prerequisites

Familiarity with the basic concepts of logic and probability theory.
Knowledge of basic computer science principles and skills.
Knowledge of basic artificial intelligence problems and principles (at least at the level of CS440, and preferably at the level of CS498ea).
Mathematical ability and the ability to understand and analyze fairly complicated algorithms and data structures. (CS473 is sufficient but not necessary.)
Independence.

To catch up with knowledge required for the class, please read [Rusell & Norvig '03] (2nd edition) chapters 1-15, and [Cormen, Lasersson & Rivest '90] (or the newer edition).

Machinery: Tasks and Coursework

The course will consist of lectures by the teacher, lectures by students, and a final project, with the actual consistency depending on the number of students in the class. Due dates and recommended projects and papers for presentation are indicated in the syllabus.

Presentations by students

I expect students to present one or two papers during the course of the semester. They will select their paper/s from the list of recommended papers that is provided in the papers section of the syllabus. If a student wishes to present a paper that does not appear in this list, an approval from me (the instructor) will be required.

Final project

Students will also be expected to submit an initial project proposal and a more detailed proposal (after the first proposal was approved), according to the dates in the syllabus. Projects may have an implementation emphasis, a theoretical emphasis, or a combination (preferred, but not mandatory). I urge students to propose their own projects, but if they cannot find ones by themselves, then they may select projects from the project list provided in the projects section of the syllabus. In either case, the students should expect to meet with the instructor to present their initial proposal and their detailed proposal. Projects may be done by single students or in pairs. In the case of a pair of students, the project must be appropriately larger or more ambitious, and the contribution of each of the students clearly identifiable. In the conclusion of the class the students will present their projects in a poster session, and will also submit a technical paper describing the project (see syllabus for dates).

The proposals and the final paper must be handed in during the class on the date indicated in the syllabus. Recognizing that students may face unusual circumstances and require some flexibility in the course of the quarter, each student will have a total of seven free late (calendar) days to use as s/he sees fit. Once these late days are exhausted, any paper or proposal turned in late will be penalized at the rate of 20% per late day (or fraction thereof). Under no circumstances will an assignment be accepted more than a week after its due date. Late days are from noon to noon.

Late assignments should be turned in at the turn-in box outside my office. You must write the time and date of submission on the assignment. Alternatively, you can fax it to the course secretary (see the fax number above) or email it to eyal@cs.uiuc.edu.

The final grade will be calculated using the following formula:
0.25*technical paper + 0.25*poster + 0.3*presentation + 0.1*proposal1 + 0.1*proposal2
Up to 3% extra credit may be awarded for class participation. Bonuses will be given for more ambitious, imaginative, and original work, but standard works that are correct and sufficient will still not be penalized and will receive full credit.

Communication

I strongly encourage students to come to office hours instead of emailing questions to me. Also, as explained above, late assignments should be turned in at the turn-in box outside my office. You can also email these assignments to eyal@cs.uiuc.edu.

Please do not e-mail me with grading questions. If you want me to explain why I took points off, you can talk to me after class or during office hours. If you want a regrade, please write an explanation and hand the assignment and the explanation to me during office hours or after class.

Occasionally I may need to broadcast a message to entire class. I will do so over the WebBoard for the class.

Textbooks and Papers Information

The primary reading materials will be book chapters and papers as described in the syllabus. The main books people can use for this class are (each covers different material)

Michael Littman, Algorithms for Sequential Decision Making. 1996. Ph. D. thesis, Department of Computer Science, Brown University.

Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: an introduction. 1998, MIT Press.

Dimitri P. Bertsekas and John Tsitsiklis, Neuro-Dynamic Programming. 1996, Athena Scientific.

Stuart Russell and Peter Norvig, Artificial Intelligence, a Modern Approach, Prentice Hall, 2nd ed., 2003.

Useful Links and Resources

Reid Simmons's class on AI planning

Son Tran's class on AI planning

Comments to Eyal Amir