Computer Science – Artificial Intelligence
Scientific paper
2011-06-09
Journal Of Artificial Intelligence Research, Volume 17, pages 1-33, 2002
Computer Science
Artificial Intelligence
Scientific paper
10.1613/jair.935
Recent trends in planning research have led to empirical comparison becoming commonplace. The field has started to settle into a methodology for such comparisons, which for obvious practical reasons requires running a subset of planners on a subset of problems. In this paper, we characterize the methodology and examine eight implicit assumptions about the problems, planners and metrics used in many of these comparisons. The problem assumptions are: PR1) the performance of a general purpose planner should not be penalized/biased if executed on a sampling of problems and domains, PR2) minor syntactic differences in representation do not affect performance, and PR3) problems should be solvable by STRIPS capable planners unless they require ADL. The planner assumptions are: PL1) the latest version of a planner is the best one to use, PL2) default parameter settings approximate good performance, and PL3) time cut-offs do not unduly bias outcome. The metrics assumptions are: M1) performance degrades similarly for each planner when run on degraded runtime environments (e.g., machine platform) and M2) the number of plan steps distinguishes performance. We find that most of these assumptions are not supported empirically; in particular, that planners are affected differently by these assumptions. We conclude with a call to the community to devote research resources to improving the state of the practice and especially to enhancing the available benchmark problems.
Dahlman E.
Howe A. E.
No associations
LandOfFree
A Critical Assessment of Benchmark Comparison in Planning does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.
If you have personal experience with A Critical Assessment of Benchmark Comparison in Planning, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and A Critical Assessment of Benchmark Comparison in Planning will most certainly appreciate the feedback.
Profile ID: LFWR-SCP-O-580279