Information Bottleneck Problem Revisited

Document Type

Conference Proceeding

Publication Date



In this paper, we revisit the information bottleneck problem whose formulation and solution are of great importance in both information theory and statistical learning applications. We go into details as to why the problem was first introduced and how the algorithm proposed using Lagrangian method to solve such problems fell short of an exact solution. We then revisit the limitations of such Lagrangian methods, and propose to adopt a more systematic method, namely, Alternate Direction Method of Multipliers (ADMM) to develop a more efficient ADMM algorithm with randomized permutation orders to solve such problems. More importantly, we mathematically demonstrate how our suggested method outperforms the original Information Bottleneck (IB) method. At the end, we provide numerical results to demonstrate the notable advantages our algorithm attains as compared with the well-known IB approach in terms of both attained objective function values and the resulting constraints. We further inspect the concepts of accuracy and convergence and the trade-off between them in our method.

Publication Source (Journal or Book title)

2019 57th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2019

First Page


Last Page


This document is currently not available here.