Intro
Foreword
Preface
Contents
Acronyms
1 Information and Coding
1.1 Information, Probability, and Coding
1.1.1 What is Information?
1.1.2 Prefix Coding
1.1.3 Kraft Inequality
1.2 Shannon Information
1.2.1 Probability Distribution
1.2.2 Shannon's Coding Theorem
1.2.3 Shannon Information
1.3 Universal Coding
1.3.1 Two-part Coding
1.3.2 Bayes Coding
1.3.3 Counting Coding
1.3.4 Normalized Maximum Likelihood Coding
1.3.5 Kolmogorov Complexity
1.4 Stochastic Complexity
1.4.1 Stochastic Complexity for Parametric Classes

1.4.2 Shtarkov's Min-Max Regret
1.4.3 Generalized Coding Theorem
1.5 Parametric Complexity
1.5.1 Asymptotic Approximation Method
1.5.2 g-Function-Based Method*
1.5.3 Fourier Method*
1.5.4 Combinatorial Method
1.5.5 Monte Carlo Method
1.6 MDL Principle
1.6.1 Machine Learning with MDL Principle
1.6.2 Estimation
1.6.3 Prediction
1.6.4 Testing
1.7 Summary of This Chapter
References
2 Parameter Estimation
2.1 Maximum Likelihood Estimation
2.1.1 Maximum Likelihood Estimator
2.1.2 MLE for Multivariate Gaussian and Outlier Detection

2.1.3 MLE for Linear Regression
2.1.4 Properties of Maximum Likelihood Estimator
2.2 EM Algorithm
2.2.1 EM Algorithm for Latent Variable Models
2.2.2 Incremental EM Algorithm for Online Outlier Detection
2.3 Maximum a Posteriori Estimation
2.3.1 MAP Estimation and Regularization
2.3.2 Sparse Regularized Linear Regression
2.3.3 Sparse Regularized Graphical Model*
2.4 Gradient Descent Methods
2.4.1 Gradient Descent Algorithms
2.5 High-Dimensional Penalty Selection
2.5.1 Luckiness Normalized Maximum Likelihood Code-length
2.5.2 Penalty Selection with LNML*

2.5.3 Analytical Bounds for LNML Code-length*
2.6 Bayesian Estimation
2.6.1 Bayesian Estimator
2.6.2 Gibbs Sampler
2.7 Summary of This Chapter
References
3 Model Selection
3.1 Model Selection
3.1.1 Problem Setting
3.1.2 Akaike's Information Criterion
3.1.3 Bayesian Information Criterion
3.1.4 Minimum Message Length Criterion
3.1.5 Cross-Validation
3.2 Minimum Description Length Criterion
3.2.1 MDL Criterion
3.2.2 Consistency
3.2.3 Estimation Optimality*
3.2.4 Rate of Convergence
3.2.5 Sequential Normalized Maximum Likelihood Criterion

3.3 Applications of MDL Criterion
3.3.1 Histogram Density Estimation
3.3.2 Non-negative Matrix Factorization*
3.3.3 Decision Tree Learning
3.3.4 Dimensionality Selection for Word Embedding
3.3.5 Time Series Model Selection
3.3.6 Multivariate Linear Regression*
3.4 Summary of This Chapter
References
4 Latent Variable Model Selection
4.1 MDL Approach to Latent Variable Model Selection
4.1.1 Non-identifiability for Latent Variable Models
4.2 Latent Stochastic Complexity
4.2.1 LSC Criterion
4.2.2 Computational Complexity of LSC