pytagi.hsm_calibration#
Post-hoc gain calibration of the hierarchical softmax output.
A hierarchical softmax classifier reads its class probability off a binary tree. At node h on the path of class k, the network output Z_h gives the branch probability P_h = Phi(s_h * G_h * Z_h), where Phi is the standard normal CDF, s_h = +1 or -1 is the branch sign and G_h is a readout gain. The class probability is the product of P_h over the path.
A network trained with a small observation noise is accurate but overconfident. This module keeps the trained network fixed and treats each gain G_h as a Gaussian random variable. It updates the gains on a held-out split with an approximate Bayesian update, using the observation “the branch of the true class was taken” with the Bernoulli variance mu_P (1 - mu_P).
The mean and the variance of P_h use Gaussian moment matching of the product G_h * Z_h. The covariance of G_h and P_h is var_G * d mu_P / d mu_G (Stein’s lemma applied to the moment-matched mean), which agrees with Monte Carlo better than the covariance of a joint Gaussian approximation of (G_h, G_h Z_h). The update is therefore a scalar recursion per node that needs the forward-pass means and variances only. Neither the training loop nor the C++/CUDA backend is involved.
The fixed readout of the library is this model with G_h = 3 and no gain uncertainty, so the default prior (mean 3) starts from the fixed readout.
When the number of classes is not a power of two, the tree has missing leaves. The predictive class probabilities are then normalized over the classes, while the calibration update uses each node of the path of the true class on its own, so its objective differs slightly from the normalized predictive.
Typical use:
calibrator = HSMGainCalibrator(num_classes=10, sharing="level")
calibrator.calibrate(mu_val, var_val, labels_val)
prob, prob_var = calibrator.class_probabilities(mu_test, var_test)
Attributes#
Classes#
Gaussian readout gains for the hierarchical softmax. |
Functions#
|
|
|
|
|
Variance of Phi(X) for a Gaussian X with variance |
|
Means of both branches, Phi(a) and Phi(-a), and the variance of |
|
Moments of the branch probability P = Phi(sign * G * Z). |
Module Contents#
- pytagi.hsm_calibration._phi_variance(a: numpy.ndarray, var: numpy.ndarray) numpy.ndarray[source]#
Variance of Phi(X) for a Gaussian X with variance
var.ais the standardized mean mu_X / sqrt(1 + var). The variance is the integral of exp(-a^2 (1 + t^2) / 2) / (1 + t^2) over t in [b, 1], divided by pi, with b = 1 / sqrt(1 + 2 var). It is evaluated with an 8-point Gauss-Legendre rule, which stays positive and avoids the cancellation of the closed form with Owen’s T function for smallvar.
- pytagi.hsm_calibration._branch_mean_var(mu_z, var_z, mu_gain, var_gain, with_variance=False)[source]#
Means of both branches, Phi(a) and Phi(-a), and the variance of Phi(G * Z): the part of
_branch_momentsthat prediction needs.The smaller of the two means is evaluated directly, so a branch probability in the far tail does not round to zero as 1 - Phi(a) would.
- Returns:
(mu_plus, mu_minus, var);varis None unlesswith_variance.
- pytagi.hsm_calibration._branch_moments(mu_z, var_z, mu_gain, var_gain, sign, with_variance: bool = False)[source]#
Moments of the branch probability P = Phi(sign * G * Z).
G and Z are independent Gaussians. The mean and the variance of P replace the product S = G * Z by a Gaussian with its exact mean and variance, which is exact when either G or Z is deterministic. The covariance is var_G * d mu_P / d mu_G, from Stein’s lemma.
- Returns:
The mean of P, the covariance of G and P, and the variance of P (None unless
with_variance).
- class pytagi.hsm_calibration.HSMGainCalibrator(num_classes: int, sharing: str = 'level', mu_gain_init: float = 3.0, sigma_gain_init: float = 1.0, min_mu_gain: float = 0.001, eps: float = 1e-06)[source]#
Gaussian readout gains for the hierarchical softmax.
The branch probability Phi(s * G * Z) has its mean and variance from Gaussian moment matching of G * Z, and its covariance with G from Stein’s lemma (see the module documentation).
- Parameters:
num_classes (int) – The number of classes.
sharing (str) –
"node"for one gain per tree node,"level"for one gain per tree level, or"global"for a single gain.mu_gain_init (float) – Prior mean of the gains. The default 3 is the gain of the fixed readout (
Utils.obs_to_label_prob), so the prior is “no change from the fixed readout”.sigma_gain_init (float) – Prior standard deviation of the gains.
min_mu_gain (float) – Lower bound of the gain mean. A negative gain would swap the two branches of a node.
eps (float) – The branch probability is clipped to [eps, 1 - eps] before the update to keep the Bernoulli variance positive.
- _build_tree() None[source]#
Derive the tree from the class paths.
_levelslists, for every level below the root, the units on it with their parent and the branch taken from the parent. This gives the probability of every path prefix with one pass over the units instead of one pass over the (class, level) pairs._raggeddescribes the units whose subtree is missing some class, the only ones that make the class probabilities sum to less than one.
- calibrate(mu_z: numpy.ndarray, var_z: numpy.ndarray, labels: Sequence[int], batch_size: int = 4096, reset_variance: bool = True) None[source]#
Update the gains with one pass over a held-out split.
Use data that was not used to train the network.
The samples are processed in order, as a sequence of Bayesian updates. Inside a mini-batch, updates of different gains do not depend on each other, so they are evaluated together with array operations. The result does not depend on
batch_size.- Parameters:
mu_z (np.ndarray) – Output means of the network, shape (N, len) or flat.
var_z (np.ndarray) – Output variances of the network, same shape.
labels (Sequence[int]) – The true class of each of the N samples, integers in [0, num_classes).
batch_size (int) – Samples gathered at a time. Only the units on the path of the label are read, so float32 outputs are not copied.
reset_variance (bool) – Reset the gain variance to the prior first. Pass False to stream a split through several calls.
- Raises:
ValueError – If the shapes or the labels are invalid, or if an output read on the path of a label is not finite or has a negative variance.
- _check_labels(labels, num_rows: int) numpy.ndarray[source]#
Labels as an int64 array of length
num_rowsin range.
- _update_gains(gain, mu_z, var_z, sign) None[source]#
Bayesian update of distinct gains, one observation each.
- _update_gain_scalar(gain, mu_z, var_z, sign) None[source]#
Same as
_update_gainsfor one gain, without numpy overhead.
- class_probabilities(mu_z: numpy.ndarray, var_z: numpy.ndarray, compute_var: bool = True, classes: Sequence[int] | None = None, num_threads: int = 1) Tuple[numpy.ndarray, numpy.ndarray | None][source]#
Class probabilities computed with the calibrated gains.
The probabilities sum to one when every leaf of the tree is a class. Otherwise (for example when the number of classes is not a power of two) they are normalized over the classes: p_k = u_k / M, where u_k is the product over the path of class k and M the sum over the classes. The variance treats the tree nodes as independent. For normalized probabilities it is the first-order delta method, (var(u_k) - 2 p_k cov(u_k, M) + p_k^2 var(M)) / E[M]^2, which keeps the covariance of u_k with M (the plain var(u_k) / E[M]^2 overestimates the variance of the most probable classes).
With a large tree, ask only for the classes you need: the cost is then proportional to the number of requested classes instead of the size of the tree for the paths, and to the number of incomplete subtrees for the normalization. See also
label_probabilitiesandtop_k_probabilities.- Parameters:
mu_z (np.ndarray) – Output means of the network, shape (N, len) or (len,).
var_z (np.ndarray) – Output variances of the network, same shape.
compute_var (bool) – Also return the variance of the class probability.
classes (Sequence[int]) – Classes to return, shape (C,) for all samples or (N, C) for one list per sample. Default: all classes.
num_threads (int) – Threads that process blocks of rows in parallel. The result does not depend on it.
- Returns:
The mean and the variance of the class probabilities, each of shape (N, num_classes) or (N, C). The variance is None if
compute_varis False.- Return type:
tuple
- label_probabilities(mu_z: numpy.ndarray, var_z: numpy.ndarray, labels: Sequence[int], compute_var: bool = False) Tuple[numpy.ndarray, numpy.ndarray | None][source]#
Probability of the given class of each sample, shape (N,).
This is what the negative log-likelihood needs. It does not build the (N, num_classes) matrix. Same values as
class_probabilities.- Raises:
ValueError – If
labelsare not N integers in [0, num_classes).
- top_k_probabilities(mu_z: numpy.ndarray, var_z: numpy.ndarray, k: int, num_threads: int = 1) Tuple[numpy.ndarray, numpy.ndarray][source]#
The
kmost probable classes of each sample.The result is exact. The full matrix is built a few rows at a time and not kept.
- Parameters:
k (int) – Number of classes to return, 1 <= k <= num_classes.
num_threads (int) – Passed to
class_probabilities.
- Returns:
The probabilities (N, k) in decreasing order and the class indices (N, k).
- _all_classes(mu_z, var_z, compute_var)[source]#
All classes: path products by one pass over the tree units.
- _some_classes(mu_z, var_z, classes, compute_var)[source]#
Requested classes only: gather the units on their paths.
- _normalize(mu_z, var_z, mu_u, var_u, mu_path, mu_other, var_path, siblings)[source]#
Normalized probabilities p_k = u_k / M and their variance.
The variance is the first-order delta method, (var(u_k) - 2 p_k cov(u_k, M) + p_k^2 var(M)) / E[M]^2, with the nodes independent. cov(u_k, M) = E[u_k M] - E[u_k] E[M], and E[u_k M] is a recursion up the path of class k: at a node where the path takes the branch B, E[B^2] E[u M] of the child on the path plus E[B B’] E[u] E[M] of the two children, where B’ = 1 - B is the other branch.
mu_path,mu_otherandvar_pathhave shape (…, levels);siblingsgives the column of the other child subtree of each node in the table of_subtree_mass.
- _subtree_mass(mu_z, var_z, second_moment=False)[source]#
Sum M of the class probabilities, without visiting every class.
Subtrees that contain all their classes sum to one. The sum of an incomplete subtree is P * (sum of the + subtree) + (1 - P) * (sum of the - subtree), evaluated from the deepest level up, with 1 - P computed as Phi(-a).
- Returns:
E[M] (N,); with
second_moment, also the table of E[M] of every incomplete subtree (followed by the columns 0 and 1 for a missing and a complete subtree) and E[M^2] (N,).