MDL Estimation for Small Sample Sizes and Its Application to Linear Regression

In this paper we examine the problem of performing Minimum Description Length (MDL) based estimation for small sample sizes. By this we mean cases where asymptotic forms are not appropriate. The path to our solution is embedded in a careful examination of the issue of completeness in MDL-based estimation. We develop specific MDL-based code-length formulas for use as objective functions in linear regression (fitting functions linear in the adjustable parameters). The problem we address is regression where we not only want to find the best fit for a given model order/complexity (e.g. polynomial degree), but also the best order. The accuracy of this fitting process will be assessed by how well the fitted function matches the ``true'' underlying function in simulation experiments. We succeed in obtaining several non-asymptotic expressions (code-length formulas) for the problem we consider and these results are compared with each other and with the ``classic'' asymptotic (k/2) log n MDL formula in fairly extensive simulation curve-fitting experiments. All the non-asymptotic formulas do better than (k/2) log n. Index Terms: Minimum Description Length principle, Stochastic Complexity, order estimation, model selection, statistical estimation, linear regression.

By: Byron E. Dom

Published in: RJ10030 in 1996


This Research Report is available. This report has been submitted for publication outside of IBM and will probably be copyrighted if accepted for publication. It has been issued as a Research Report for early dissemination of its contents. In view of the transfer of copyright to the outside publisher, its distribution outside of IBM prior to publication should be limited to peer communications and specific requests. After outside publication, requests should be filled only by reprints or legally obtained copies of the article (e.g., payment of royalties). I have read and understand this notice and am a member of the scientific community outside or inside of IBM seeking a single copy only.

Questions about this service can be mailed to .