Reference

Binary(θ)

Layer with binary units, with external fields θ.

CenteredRBM(visible, hidden, w)

Creates a centered RBM, with offsets initialized to zero.

CenteredRBM(rbm, λv, λh)

Creates a centered RBM, with offsets λv (visible) and λh (hidden). See http://jmlr.org/papers/v17/14-237.html for details. The resulting model is not equivalent to the original rbm, unless λv = 0 and λh = 0.

Gaussian(θ, γ)

Gaussian layer, with location parameters θ and scale parameters γ. The energy of a layer with units $h_\mu$ is $E = \sum_\mu U(h_\mu)$, with the unit potential:

\[U(h) = \frac{|\gamma|}{2} h^2 - \theta h\]

where $\theta$, $\gamma$ are the entries of θ, γ for the corresponding unit, and $h$ takes values in $\mathbb{R}$.

Potts(θ)

Layer with Potts units, with external fields θ. Encodes categorical variables as one-hot vectors. The number of classes is the size of the first dimension.

PottsGumbel(; θ)

Like Potts, but uses the Gumbel-softmax trick for GPU-friendly sampling.

RBM{V,H,W}

RBM, with visible layer of type V, hidden layer of type H, and weights of type W.

RBM(centered_rbm::CenteredRBM)

Returns an (uncentered) RBM which neglects the offsets of centered_rbm. The resulting model is not equivalent to the original centered_rbm. To construct an equivalent model, use the function uncenter(centered_rbm) instead (see uncenter). Shares parameters with centered_rbm.

ReLU(θ, γ)

Layer with ReLU units, with location parameters θ and scale parameters γ. The energy of a layer with units $h_\mu$ is $E = \sum_\mu U(h_\mu)$, with the unit potential:

\[U(h) = \frac{|\gamma|}{2} h^2 - \theta h \qquad (h \ge 0)\]

where $\theta$, $\gamma$ are the entries of θ, γ for the corresponding unit. Units are constrained to non-negative values ($U(h) = \infty$ for $h < 0$).

Spin(θ)

Layer with spin units, with external fields θ. The energy of a layer with units $s_i$ is given by:

\[E = -\sum_i \theta_i s_i\]

where each spin $s_i$ takes values $\pm 1$.

StandardizedRBM{V,H,W,Ov,Oh,Sv,Sh}

RBM with standardized layer activations. Like CenteredRBM it subtracts the offsets offset_v, offset_h from the visible and hidden activations entering the interaction, and additionally divides them by the scales scale_v, scale_h. The energy is

\[E(v,h) = E_v(v) + E_h(h) - \sum_{i\mu} w_{i\mu} \frac{v_i - \lambda_i}{\sigma_i} \frac{h_\mu - \lambda_\mu}{\sigma_\mu}\]

where $\lambda$ are the offsets and $\sigma$ the scales. A CenteredRBM is the special case with unit scales. See http://jmlr.org/papers/v17/14-237.html.

StandardizedRBM(rbm, offset_v, offset_h, scale_v, scale_h)

Creates a standardized RBM, with offsets offset_v, offset_h and scales scale_v, scale_h. The resulting model is not equivalent to the original rbm, unless the offsets are zero and the scales are one. To construct an equivalent model instead, use standardize.

StandardizedRBM(rbm)

Creates a standardized RBM from rbm, with offsets initialized to zero and scales to one (so the constructed model is equivalent to rbm).

dReLU(; θp, θn, γp, γn)

Double ReLU layer, with separate parameters for positive and negative parts. The energy of a layer with units $h_\mu$ is $E = \sum_\mu U(h_\mu)$, with the unit potential:

\[U(h) = \begin{cases} \frac{|\gamma^+|}{2} h^2 - \theta^+ h & h \ge 0 \\[4pt] \frac{|\gamma^-|}{2} h^2 - \theta^- h & h < 0 \end{cases}\]

where $\theta^+, \theta^-, \gamma^+, \gamma^-$ are the entries of θp, θn, γp, γn for the corresponding unit, and $h$ takes values in $\mathbb{R}$.

nsReLU(; θ, Δ, ξ)

A variant of xReLU units without scale parameter γ (which is fixed at 1). This is done to remove the gauge invariance between the weights and the hidden units scale. The energy of a layer with units $h_\mu$ is $E = \sum_\mu U(h_\mu)$, with the unit potential:

\[U(h) = \begin{cases} \frac{1}{2(1+\eta)} h^2 - \left(\theta + \frac{\Delta}{1+\eta}\right) h & h \ge 0 \\[4pt] \frac{1}{2(1-\eta)} h^2 - \left(\theta - \frac{\Delta}{1-\eta}\right) h & h < 0 \end{cases} \qquad \eta = \frac{\xi}{1 + |\xi|}\]

where $\theta, \Delta, \xi$ are the entries of θ, Δ, ξ for the corresponding unit. This is the xReLU potential with $\gamma = 1$.

pReLU(; θ, γ, Δ, η)

A different parameterization of the dReLU layer, with shared scale and asymmetry ratio. The energy of a layer with units $h_\mu$ is $E = \sum_\mu U(h_\mu)$, with the unit potential:

\[U(h) = \begin{cases} \frac{|\gamma|}{2(1+\eta)} h^2 - \left(\theta + \frac{\Delta}{1+\eta}\right) h & h \ge 0 \\[4pt] \frac{|\gamma|}{2(1-\eta)} h^2 - \left(\theta - \frac{\Delta}{1-\eta}\right) h & h < 0 \end{cases}\]

where $\theta, \gamma, \Delta, \eta$ are the entries of θ, γ, Δ, η for the corresponding unit. This is the dReLU potential with $\theta^\pm = \theta \pm \Delta / (1 \pm \eta)$ and $\gamma^\pm = \gamma / (1 \pm \eta)$.

Every value of η must be finite and lie strictly inside (-1, 1). For unconstrained learned asymmetry, use xReLU or the fixed-scale nsReLU instead.

xReLU(; θ, γ, Δ, ξ)

Extended ReLU layer, like pReLU but with unbounded asymmetry parameter. The energy of a layer with units $h_\mu$ is $E = \sum_\mu U(h_\mu)$, with the unit potential:

\[U(h) = \begin{cases} \frac{|\gamma|}{2(1+\eta)} h^2 - \left(\theta + \frac{\Delta}{1+\eta}\right) h & h \ge 0 \\[4pt] \frac{|\gamma|}{2(1-\eta)} h^2 - \left(\theta - \frac{\Delta}{1-\eta}\right) h & h < 0 \end{cases} \qquad \eta = \frac{\xi}{1 + |\xi|}\]

where $\theta, \gamma, \Delta, \xi$ are the entries of θ, γ, Δ, ξ for the corresponding unit. This is the pReLU potential with the bounded asymmetry $\eta \in (-1, 1)$ reparameterized through the unbounded $\xi \in \mathbb{R}$.

BinaryRBM(a, b, w)
BinaryRBM(N, M)

Construct an RBM with binary visible and hidden units, which has an energy function:

\[E(v, h) = -a'v - b'h - v'wh\]

Equivalent to RBM(Binary(a), Binary(b), w).

BinaryStandardizedRBM(a, b, w, offset_v, offset_h, scale_v, scale_h)
BinaryStandardizedRBM(a, b, w)

Construct a standardized RBM with Binary visible and hidden layers. With the short form the offsets are zero and the scales one (equivalent to the plain BinaryRBM).

CenteredBinaryRBM(a, b, w, λv = 0, λh = 0)

Construct a centered binary RBM. The energy function is given by:

\[E(v,h) = -a' * v - b' * h - (v - λv)' * w * (h - λh)\]

GaussianRBM(θv, γv, θh, γh, w)

Construct an RBM with Gaussian visible and hidden units. Equivalent to RBM(Gaussian(θv, γv), Gaussian(θh, γh), w).

HopfieldRBM(g, θ, γ, w)
HopfieldRBM(g, w)

Construct an RBM with spin visible units and Gaussian hidden units. If not given, θ = 0 and γ = 1 by default.

\[E(v, h) = -g'v - θ'h + \sum_\mu \frac{γ_\mu}{2} h_\mu^2 - v'wh\]

SpinRBM(a, b, w)

Construct an RBM with spin visible and hidden units. Equivalent to RBM(Spin(a), Spin(b), w).

SpinStandardizedRBM(a, b, w, offset_v, offset_h, scale_v, scale_h)
SpinStandardizedRBM(a, b, w)

Construct a standardized RBM with Spin visible and hidden layers. With the short form the offsets are zero and the scales one (equivalent to the plain SpinRBM).

ais(rbm0, rbm1, v0, βs)

Provided v0 is an equilibrated sample from rbm0, returns F such that mean(exp.(F)) is an unbiased estimator of Z1/Z0, the ratio of partition functions of rbm1 and rbm0.

!!! tip Use logmeanexp logmeanexp(F), using the function logmeanexp from LogStatFunctions.jl, tends to give a better approximation of log(Z1) - log(Z0) than mean(F).

aise(rbm, [βs]; [nbetas], init=rbm.visible, nsamples=1)

AIS estimator of the log-partition function of rbm. It is recommended to fit init to the single-site statistics of rbm (or the data).

!!! tip Use large nbetas For more accurate estimates, use larger nbetas. It is usually better to have large nbetas and small nsamples, rather than large nsamples and small nbetas.

anneal(rbm0, rbm1; β)

Returns an RBM that interpolates between rbm0 and rbm1. Denoting by E0(v, h) and E1(v, h) the energies assigned by rbm0 and rbm1, respectively, the returned RBM assigns energies given by:

E(v,h) = (1 - β) * E0(v) + β * E1(v, h)

batch_size(rbm, v, h)

Returns the batch size if energy(rbm, v, h) were computed.

batch_size(layer, x)

Batch sizes of x, with respect to layer.

batchcov(layer, x; [wts], [mean])

Covariance of x over batch dimensions, weigthed by wts.

batchdims(layer, x)

Indices of batch dimensions in x, with respect to layer.

batchmean(layer, x; [wts])

Mean of x over batch dimensions, weigthed by wts.

batchmean_moments(layer, moments; [wts])

Average a per-configuration moments array (as returned by moments_from_inputs with batched inputs) over its batch dimensions, weighted by wts (lazy uniform weights by default).

batchstd(layer, x; [wts], [mean])

Standard deviation of x over batch dimensions, weigthed by wts.

batchvar(layer, x; [wts], [mean])

Variance of x over batch dimensions, weigthed by wts.

categorical_rand(ps)

Randomly draw i with probability ps[i]. You must ensure that ps defines a proper probability distribution.

categorical_sample(P)

Given a probability array P of size (q, *), returns an array C of size (*), such that C[i] ∈ 1:q is a random sample from the categorical distribution P[:,i]. You must ensure that P defines a proper probability distribution.

categorical_sample_from_logits(logits)

Given a logits array logits of size (q, *) (where q is the number of classes), returns an array X of size (*), such that X[i] is a categorical random sample from the distribution with logits logits[:,i].

categorical_sample_from_logits_gumbel(logits)

Like categoricalsamplefrom_logits, but using the Gumbel trick.

center!(centered_rbm, offset_v = 0, offset_h = 0)

Transforms the offsets of centered_rbm. The transformed model is equivalent to the original one (energies differ by a constant).

center(rbm::RBM, offset_v = 0, offset_h = 0)

Constructs a CenteredRBM equivalent to the given rbm. The energies assigned by the two models differ by a constant amount,

\[E(v,h) - E_c(v,h) = \sum_{i\mu}w_{i\mu}\lambda_i\lambda_\mu\]

where $E(v,h)$ is the energy assigned by the original rbm, and $E_c(v,h)$ is the energy assigned by the returned CenteredRBM.

This is the inverse operation of uncenter.

To construct a CenteredRBM that simply includes these offsets, call CenteredRBM(rbm, offset_v, offset_h) instead.

center_from_data!(rbm::CenteredRBM, data; [wts])

Sets the visible and hidden offsets from the means of data. The model is unchanged (energies differ by a constant).

center_hidden_from_data!(rbm::CenteredRBM, data; [wts], damping = 1)

Sets the hidden offsets to the mean hidden activations conditioned on data. The model is unchanged (energies differ by a constant).

center_visible_from_data!(rbm::CenteredRBM, data; [wts])

Sets the visible offsets to the mean of data. The model is unchanged (energies differ by a constant).

cgf(layer, [inputs])

Cumulant generating function of layer, reduced over layer dimensions.

cold_metropolis(rbm, v; steps = 1)

Samples the rbm at zero temperature, starting from configuration v.

collect_states(layer)

Returns an array of all states of layer. Only defined for discrete layers.

colors(layer)

Number of possible states of units in discrete layers.

delta_energy(rbm)

Compute the (constant) energy shift with respect to the equivalent normal RBM.

energies(layer, x)

Energies of units in layer (not reduced over layer dimensions).

energy(rbm, v, h)

Energy of the rbm in the configuration (v,h).

energy(layer, x)

Layer energy, reduced over layer dimensions.

flatten(layer, x)

Returns a vectorized version of x.

free_energy(rbm, v)

Free energy of visible configuration (after marginalizing hidden configurations).

generate_sequences(n, A = 0:1)

Retruns an iterator over all sequences of length n out of the alphabet A.

gumbel_to_potts(rbm)

Converts PottsGumbel layers to Potts layers.

infinite_minibatches(ds...; batchsize, shuffle = true)

Infinite iterator over shuffled minibatches of batchsize observations from ds.

initialize!(rbm, [data]; ϵ = 1e-6)

Initializes the RBM and returns it. If provided, matches average visible unit activities from data.

initialize!(layer, [data]; ϵ = 1e-6)

Initializes a layer and returns it. If provided, matches average unit activities from data.

initialize_w!(rbm, data; λ = 0.1)

Initializes rbm.w such that typical inputs to hidden units are λ.

inputs_h_from_v(rbm, v)

Interaction inputs from visible to hidden layer.

inputs_v_from_h(rbm, h)

Interaction inputs from hidden to visible layer.

interaction_energy(rbm, v, h)

Weight mediated interaction energy.

log_likelihood(rbm, v)

Log-likelihood of v under rbm, with the partition function compued by extensive enumeration. For discrete layers, this is exponentially slow for large machines.

log_partition(rbm)

Log-partition of rbm, computed by extensive enumeration of visible states (except for particular cases such as Gaussian-Gaussian RBM). This is exponentially slow for large machines.

For Gaussian-Gaussian RBMs, the exact Gaussian integral is used when the joint precision matrix (with Gaussian precisions abs.(γ)) is positive definite. Non-normalizable models with a singular or indefinite joint precision return Inf.

If your RBM has a smaller hidden layer, consider mirroring the layers of the rbm first (see mirror).

log_partition_zero_weight(rbm)

Log-partition function of a zero-weight version of rbm.

log_pseudolikelihood(rbm, v; exact = false)

Log-pseudolikelihood of v. If exact is true, the exact pseudolikelihood is returned. But this is slow if v consists of many samples. Therefore by default exact is false, in which case the result is a stochastic approximation, where a random site is selected for each sample, and its conditional probability is calculated. In average the results with exact = false coincide with the deterministic result, and the estimate is more precise as the number of samples increases.

log_pseudolikelihood_exact(rbm, v)

Log-pseudolikelihood of v. This function computes the exact pseudolikelihood, doing traces over all sites. Note that this can be slow for large number of samples. Implemented for Binary, Spin, Potts, and PottsGumbel visible layers.

log_pseudolikelihood_sites(rbm, v, sites)

Log-pseudolikelihood of a site conditioned on the other sites, where sites is an array of site indices (CartesianIndex), one for each sample. Returns an array of log-pseudolikelihood values, for each sample. Implemented for Binary, Spin, Potts, and PottsGumbel visible layers.

log_pseudolikelihood_stoch(rbm, v)

Log-pseudolikelihood of v. This function computes an stochastic approximation, by doing a trace over random sites for each sample. For large number of samples, this is in average close to the exact value of the pseudolikelihood.

mean_from_moments(layer, moments)

Mean unit activations <x> from a moments array (see moments_from_samples for the layout). Batch dimensions of moments are preserved.

mean_h_from_v(rbm, v)

Mean unit activation values, conditioned on the other layer, <h | v>.

mean_v_from_h(rbm, h)

Mean unit activation values, conditioned on the other layer, <v | h>.

meanvar_from_inputs(layer, [inputs])

Mean and variance of unit activations from inputs.

metropolis!(v, rbm; β = 1)

Metropolis-Hastings sampling from rbm at inverse temperature β. Uses v[:,:,..,:,1] as initial configurations, and writes the Monte-Carlo chains in v[:,:,..,:,2:end].

metropolis(rbm, v; β = 1, steps = 1)

Metropolis-Hastings sampling from rbm at inverse temperature β, starting from configuration v. Moves are proposed by normal Gibbs sampling.

mirror(rbm)

Returns a new RBM with visible and hidden layers flipped.

mode_h_from_v(rbm, v)

Mode unit activations, conditioned on the other layer.

mode_v_from_h(rbm, h)

Mode unit activations, conditioned on the other layer.

moments_from_inputs(layer, [inputs])

Moments of the unit activations under the conditional distribution given inputs, in the same layout as moments_from_samples: the first axis indexes the moment, the next axes are size(layer), and any trailing batch dimensions of inputs are preserved (the moments are per-configuration, not batch-averaged).

moments_from_samples(layer, data; [wts])

Empirical moments of data, batch-averaged with weights wts. Each layer defines which moments it computes (see the docstrings of its specific methods); generally they are the sufficient statistics of the layer distribution, which do not depend on the layer parameters, so they can be computed once from a dataset and reused as the parameters change (see pcd!). The first axis indexes the moment and the remaining axes are size(layer) (batch dimensions of data are averaged over). The number of moments need not match the number of parameters (e.g. nsReLU uses the 4-slot dReLU layout while having 3 parameters).

moments_from_inputs returns conditional moments in this same layout, and ∂energy_from_moments consumes it.

moments_from_samples(layer::Gaussian, data; [wts])

Two moment slots: <x> and <x^2>.

moments_from_samples(layer::ReLU, data; [wts])

Two moment slots: <x> and <x^2> (same as Gaussian).

moments_from_samples(layer::Union{Binary, Spin, Potts, PottsGumbel}, data; [wts])

One moment slot: <x>.

moments_from_samples(layer::Union{dReLU, pReLU, xReLU, nsReLU}, data; [wts])

Four moment slots: <xp>, <xn>, <xp^2>, <xn^2>, where xp = max(x, 0) and xn = min(x, 0).

onehot_decode(X)

Given a onehot encoded array X of N + 1 dimensions, returns the equivalent categorical array of N dimensions.

onehot_encode(A, code)

Given an array A of N dimensions, returns a one-hot encoded BitArray of N + 1 dimensions where single entries of the first dimension are one.

pcd!(rbm, data; kwargs...)

Train an RBM with Persistent Contrastive Divergence (PCD).

pcd! repeatedly draws mini-batches from data, performs steps Gibbs updates of persistent fantasy particles, estimates the positive/negative phase gradients, applies optional regularization and gauge constraints, and updates model parameters with an Optimisers.jl rule.

data must have shape (size(rbm.visible)..., nsamples).

Keyword arguments

• batchsize::Int=1: number of samples per update.
• iters::Int=1: number of parameter updates.
• wts::AbstractVector{<:Real}: finite, positive per-sample weights, lazy uniform weights by default. Zero or negative weights raise an ArgumentError — drop observations meant to be excluded (and their weights) beforehand. Callbacks receive the minibatch weights as wd.
• steps::Int=1: Gibbs steps used to update persistent chains each iteration.
• optim::AbstractRule=Adam(): optimizer rule from Optimisers.jl.
• moments=moments_from_samples(rbm.visible, data; wts): data moments used by the positive phase.
• l2_fields::Real=0: L2 regularization on visible fields.
• l1_weights::Real=0: L1 regularization on interaction weights.
• l2_weights::Real=0: L2 regularization on interaction weights.
• l2l1_weights::Real=0: group-like L2/L1 weight regularization.
• zerosum::Bool=true: enforce zero-sum gauge on Potts layers.
• rescale::Bool=true: rescale weights (mainly useful for continuous hidden units).
• callback=Returns(nothing): called after every update as callback(; rbm, optim, state, ps, iter, vd, wd, ∂, vm). Slurp unused keywords with a trailing _....
• vm: initial fantasy particles. By default, min(batchsize, nsamples) chains sampled from the visible layer with zero inputs.
• shuffle::Bool=true: whether to reshuffle samples between epochs.
• ps: optimized parameter container. By default, this contains the visible, hidden, and interaction parameters.
• state=setup(optim, ps): optimizer state.

Returns (state, ps).

potts_to_gumbel(rbm)

Converts Potts layers to PottsGumbel layers.

raise(rbm::RBM, βs; v, init)

Reverse AIS estimator of the log-partition function of rbm. While aise tends to understimate the log of the partition function, raise tends to overstimate it. v must be an equilibrated sample from rbm.

!!! tip Use logmeanexp If F = raise(...), then -logmeanexp(-F), using the function logmeanexp from LogStatFunctions.jl, tends to give a better approximation of log(Z) than mean(F).

!!! tip Sandwiching the log-partition function If Rf = aise(...), Rr = raise(...) are the AIS and reverse AIS estimators, we have the stochastic bounds logmeanexp(Rf) ≤ log(Z) ≤ -logmeanexp(-Rr).

randgumbel(T = Float64)

Generates a random Gumbel variate.

randnt([rng], a)

Random standard normal lower truncated at a (that is, Z ≥ a).

randnt_half([rng], μ, σ)

Samples the normal distribution with mean μ and standard deviation σ truncated to positive values.

reconstruction_error(rbm, v; steps = 1)

Stochastic reconstruction error of v.

rescale_activations!(layer, λ::AbstractArray)

For continuous layers with scale parameters, re-parameterizes such that unit activations are divided by λ, and returns true. For other layers, does nothing and returns false.

rescale_hidden!(rbm::CenteredRBM, λ::AbstractArray)

Scales parameters such that hidden unit activations are divided by λ, preserving the modeled distribution. This assumes the hidden units have a scale parameter, otherwise it does nothing and returns false. Since the interaction involves h - offset_h, the hidden offsets are divided by λ together with the activations.

rescale_hidden!(rbm, λ::AbstractArray)

For continuous hidden units with a scale parameter, scales parameters such that hidden unit activations are divided by λ, and returns true. For other hidden units does nothing and returns false. The modified RBM is equivalent to the original one.

rescale_hidden!(rbm::StandardizedRBM, λ::AbstractArray)

Rescale hidden unit activities by λ, which should be an array of the same size as the hidden units. This assumes the hidden units have a scale parameter, otherwise it does nothing and returns false.

rescale_hidden_activations!(rbm::StandardizedRBM)

Absorbs scale_h into the hidden layer if it has a scale parameter, returning true if this was done. The modified RBM is equivalent to the original one.

rescale_weights!(rbm)

For continuous hidden units with a scale parameter, scales parameters such that the weights attached to each hidden unit have norm 1. Hidden units whose incoming weights have zero norm are left unchanged. For a StandardizedRBM the unstandardized weights are normalized (see weight_norms): the standardized weights are invariant under rescaling of hidden unit activities, so they cannot be constrained to have unit norm.

reshape_maybe(x, shape)

Like reshape(x, shape), except that zero-dimensional outputs are returned as scalars.

sample_from_inputs(layer, [inputs])

Samples unit activations of layer conditioned on the given inputs.

sample_h_from_h(rbm, h; steps=1)

Samples a hidden configuration conditional on another hidden configuration h. Ensures type stability by requiring that the returned array is of the same type as h.

sample_h_from_v(rbm, v)

Samples a hidden configuration conditional on the visible configuration v.

sample_v_from_h(rbm, h)

Samples a visible configuration conditional on the hidden configuration h.

sample_v_from_v(rbm, v; steps=1)

Samples a visible configuration conditional on another visible configuration v. Ensures type stability by requiring that the returned array is of the same type as v.

sitedims(layer)

Number of dimensions of layer, with special handling of Potts layer, for which the first dimension doesn't count as a site dimension.

sitesize(layer)

Size of layer, with special handling of Potts layer, for which the first dimension doesn't count as a site dimension.

sqrt1half(x)

Accurate computation of sqrt(1 + (x/2)^2) + |x|/2.

standardize!(rbm::StandardizedRBM, offset_v, offset_h, scale_v, scale_h)

Transforms the offsets and scales of rbm in place. The transformed model is equivalent to the original one (energies differ by a constant). In-place analogue of standardize.

standardize(rbm, offset_v = 0, offset_h = 0, scale_v = 1, scale_h = 1)

Constructs a StandardizedRBM equivalent to the given rbm (a plain RBM or another StandardizedRBM), with the given offsets and scales. The energies assigned by the two models differ by a constant amount, so the modeled distribution is unchanged.

This is the inverse operation of unstandardize. To construct a StandardizedRBM that simply adopts these offsets and scales without preserving the distribution, call StandardizedRBM(rbm, offset_v, offset_h, scale_v, scale_h) instead.

standardize_hidden_from_v!(rbm::StandardizedRBM, v; [wts], damping = 0, ϵ = 0)

Sets the hidden offsets and scales to the mean and standard deviation of hidden unit activations conditioned on v. The model is unchanged (energies differ by a constant).

standardize_visible_from_data!(rbm::StandardizedRBM, data; [wts], ϵ = 0)

Sets the visible offsets and scales to the mean and standard deviation of data. The model is unchanged (energies differ by a constant).

std_from_inputs(layer, [inputs])

Standard deviation of unit activations from inputs.

tnmean(a)

Mean of the standard normal distribution, truncated to the interval (a, +∞).

tnmeanvar(a)

Mean and variance of the standard normal distribution truncated to the interval (a, +∞). Equivalent to tnmean(a), tnvar(a) but saves some common computations. WARNING: tnvar(a) can fail for very very large values ofa`.

tnvar(a)

Variance of the standard normal distribution, truncated to the interval (a, +∞). WARNING: Fails for very very large values of a.

total_mean_from_inputs(layer, [inputs]; [wts])

Total mean of unit activations from inputs.

total_mean_h_from_v(rbm, v; [wts])

Total mean of hidden unit activations from visible activities.

total_mean_v_from_h(rbm, h; [wts])

Total mean of visible unit activations from given hidden activities.

total_meanvar_from_inputs(layer, [inputs]; [wts])

Total mean and total variance of unit activations from inputs.

total_meanvar_h_from_v(rbm, v; [wts])

Total mean and total variance of hidden unit activations from visible activities.

total_meanvar_v_from_h(rbm, h; [wts])

Total mean and total variance of visible unit activations from hidden activities.

total_var_from_inputs(layer, [inputs]; [wts])

Total variance of unit activations from inputs.

total_var_h_from_v(rbm, v; [wts])

Total variance of hidden unit activations from given visible activities.

total_var_v_from_h(rbm, h; [wts])

Total variance of unit activations from given hidden activities.

uncenter(centered_rbm::CenteredRBM)

Constructs an RBM equivalent to the given CenteredRBM. The energies assigned by the two models differ by a constant amount,

\[E(v,h) - E_c(v,h) = \sum_{i\mu}w_{i\mu}\lambda_i\lambda_\mu\]

where $E_c(v,h)$ is the energy assigned by centered_rbm and $E(v,h)$ is the energy assigned by the RBM constructed by this method.

This is the inverse operation of center.

To construct an RBM that simply neglects the offsets, call RBM(centered_rbm) instead.

uniform_wts(layer, x)

Lazy uniform weights over the batch dimensions of x.

unstandardize(rbm)

Convert a StandardizedRBM back to an equivalent plain RBM. Note: this does not enforce zerosum gauge; call zerosum(unstandardize(rbm)) if needed.

validate_wts(wts)

Asserts that the data weights wts are finite and positive.

var_from_moments(layer, moments)

Variance of unit activations from a moments array (see moments_from_samples for the layout). Batch dimensions of moments are preserved.

var_h_from_v(rbm, v)

Variance of unit activation values, conditioned on the other layer, var(h | v).

var_v_from_h(rbm, h)

Variance of unit activation values, conditioned on the other layer, var(v | h).

weight_norms(std_rbm::StandardizedRBM)

Computes the norms of the unstandardized weights for each hidden unit. If you want the norms of the standardized weights, use weight_norms(RBM(std_rbm)).

wmean(A; [wts])

Weighted mean of A along its trailing dimensions, weighted by wts (see wsum). By default, lazy uniform weights over all of A, which reduce like an ordinary mean without allocating a weights array or promoting eltypes.

\[\frac{\sum_i A_i w_i}{\sum_i w_i}\]

wsum(A, wts)

Weighted sum of A along its trailing dimensions, weighted by wts.

\[\sum_i A_i w_i\]

The reduced dimensions are inferred from the shape of wts, which must match the trailing dimensions of A (all of A for a full reduction, returning a scalar). Reduced dimensions are dropped from the result.

zerosum!(rbm)

In-place zero-sum gauge on rbm.

zerosum!(∂, rbm)

Projects the gradient so that it doesn't modify the zerosum gauge.

zerosum!(∂, rbm::Union{CenteredRBM, StandardizedRBM})

Projects the gradient so that it doesn't modify the zerosum gauge of the equivalent plain RBM (see uncenter, unstandardize), with offsets and scales held fixed.

The gauge condition applies to the parameters of the equivalent plain RBM: for the weights it reads sum(w ./ scale_v; dims = 1) == 0 over Potts colors (and similarly for hidden Potts with scale_h), so the gradient component removed here is the corresponding gauge direction ξ .* scale_v. For a CenteredRBM the scales are one and these conditions coincide with the plain RBM ones.

zerosum!(rbm::Union{CenteredRBM, StandardizedRBM})

In-place version of zerosum(rbm). Offsets (and scales) are not modified.

zerosum(rbm::CenteredRBM)

Returns an equivalent CenteredRBM, with the same offsets, whose equivalent uncentered RBM (see uncenter) is in the zerosum gauge. Only affects Potts layers. If the rbm doesn't have Potts layers, does nothing.

Note that the gauge condition applies to the uncentered parameters: since the interaction energy involves the centered v - offset_v, sums over Potts colors of the centered weights are compensated differently than in a plain RBM.

zerosum(rbm)

Returns an equivalent rbm in zerosum gauge. Only affects Potts layers. If the rbm doesn't have Potts layers, does nothing.

zerosum(rbm::StandardizedRBM)

Returns an equivalent StandardizedRBM, with the same offsets and scales, whose equivalent unstandardized RBM (see unstandardize) is in the zerosum gauge. Only affects Potts layers. If the rbm doesn't have Potts layers, does nothing.

Note that the gauge condition applies to the unstandardized parameters: the standardized weights and fields need not sum to zero over Potts colors, because the interaction energy involves the standardized (v - offset_v) / scale_v, for which sums over colors are not constant when the offsets and scales vary across colors.

∂cgf(layer, [inputs]; [wts])

Unit activation moments, conjugate to layer parameters. These are obtained by differentiating cgfs with respect to the layer parameters. Averages over configurations (weigthed by wts).

∂cgfs(layer, [inputs])

Gradient of cgfs with respect to the layer parameters, for each configuration of inputs (batch dimensions are preserved; the first axis indexes the parameter, as in layer.par). Since the cumulant generating function and the energy are conjugate, this is -∂energy_from_moments evaluated at the conditional moments given inputs.

∂energy(layer, data; [wts])

Derivative of average energy of data with respect to layer parameters.

∂energy_from_moments(layer, moments)

Derivative of the layer's mean energy with respect to its parameters, evaluated at a moments array (see moments_from_samples for the layout). The first axis of the result indexes the parameter, as in layer.par, and trailing batch dimensions of moments are preserved. Since the energy is linear in the sufficient statistics, this is a linear map of moments, with coefficients that may depend on the current parameters.

∂free_energy(rbm, v)

Gradient of free_energy(rbm, v) with respect to model parameters. If v consists of multiple samples (batches), then an average is taken.

∂regularize!(∂, rbm; l2_fields = 0, l1_weights = 0, l2_weights = 0, l2l1_weights = 0)

Updates RBM gradients ∂, with the regularization gradient.

@declare_layer Layer (θ = zeros, γ = ones)

Declares a layer type Layer whose named parameters are the rows of a shared par array, in the given order, with the given default initializers. Generates the struct (with par size validation and a _validate_layer_parameters hook in the inner constructor), the Layer(par), keyword, Layer(T, sz), and Layer(sz) constructors, Base.propertynames, the Base.getproperty accessors returning views into par, and the _construct_like trait used by generic functions such as anneal.