New Restricted Boltzmann Machine Adds Multi-State Hidden Units for Discrete Reasoning Tasks

Researchers have developed an extension to the Gaussian-Bernoulli Restricted Boltzmann Machine (GB-RBM) that incorporates multi-valued hidden units, aiming to better capture discrete, structured representations needed for tasks like associative memory and symbolic reasoning.

The New Architecture#

The Gaussian-Multinoulli Restricted Boltzmann Machine (GM-RBM) replaces the binary hidden units found in standard RBMs with q-state categorical variables inspired by Potts models from statistical physics. This architectural change expands the possible configurations a model's latent space can represent, enabling it to encode more nuanced, discrete concepts than its binary predecessor.

Training and Implementation#

The team derived the model's energy function, conditional probability distributions, and learning algorithms from scratch. To prevent the hidden units from collapsing into a single state—a common failure mode in categorical models—they employed several practical techniques: contrastive divergence with temperature annealing and constraints that encourage diversity across categorical slots. Despite using only straightforward Gibbs sampling for updates, the approach remains computationally efficient.

Empirical Results#

The researchers evaluated GM-RBM against GB-RBM under two comparison schemes: one matching total capacity and another matching parameter count. On benchmarks testing analogical recall and structured memory tasks, GM-RBM demonstrated competitive or superior recall performance while maintaining comparable training time. The results suggest that discrete categorical hidden units can function as a practical, scalable alternative to binary latents for problems requiring discrete inference within tractable generative models.

Why It Matters#

The work clarifies when and why switching from binary to categorical hidden representations offers genuine benefits beyond simply adding more parameters—a distinction important for designing efficient neural models for symbolic and reasoning-focused applications.