Method
From white noise to structured noise masks
An input $X$ (image, video, or audio spectrogram) is split into patches and embedded as $X_p = \phi(X) \in \mathbb{R}^{N\times d}$. A masking function $\eta$ turns a noise tensor and a mask ratio $\gamma$ into a binary mask $M \in \{0,1\}^N$, and the encoder only sees the visible tokens. The only thing we change is the noise that goes into $\eta$.
Color noise, in one equation each
Given white noise $n_w \sim \mathcal{N}(0,1)$ and a $d$-dimensional Gaussian kernel $G_\sigma$, three filters give three different families of masks:
Smooth, large-scale masked regions.
Fine-grained, evenly scattered patches.
Mid-sized, spatially clustered masks.
with $\sigma_1 < \sigma_2$. These give the structured masks $M_r = \eta(X_p, n_r, \gamma)$, $M_b = \eta(X_p, n_b, \gamma)$ and $M_g = \eta(X_p, n_g, \gamma)$. Red noise keeps low frequencies and yields smooth, large-scale masked regions; blue noise keeps high frequencies and scatters visible patches finely; green noise balances both and produces mid-sized, clustered masks.
Green3D noise for video masking
Video masking has to respect both spatial coherence and temporal continuity. VideoMAE and SIGMA use random tube masking, which keeps a static mask across all frames: temporally consistent, but unrelated to the motion in the clip.
We instead apply the green-noise band-pass in 3D, using a white noise tensor $n_w$ and two 3D Gaussian kernels
$$ G_\sigma(\mathbf{x}) = \frac{1}{(2\pi)^{\frac{3}{2}}\sigma^3}\exp\left(-\frac{|\mathbf{x}|^2}{2\sigma^2}\right), \qquad \mathbf{x}=(x,y,z)\in\mathbb{R}^3 . $$
Instead of fixing $(\sigma_1,\sigma_2)$, which produces masks with limited diversity and either overly static or noisy temporal behavior, we randomly sample $(\sigma_1,\sigma_2)$ within $[0.5, 2]$ for each sequence. This generates an ensemble of 3D green noise tensors that remain in the mid-frequency range. Empirically, this stochasticity yields smoother mask evolution and avoids overfitting to a single frequency.
Green3D masks are precomputed offline as tensors (e.g. $N\times64\times64\times64$). During training they only undergo flips, normalization and resizing to the token volume (e.g. $14\times14\times8$). These are spatial-domain transforms, so the frequency properties are preserved and masking adds no computational cost at train time.
Regularized Blue Noise (R-BN) for audio masking
Audio spectrograms have a different structure. They encode time–frequency events, and clusters of visible patches, which help in vision, do not correspond to meaningful events here. Blue noise is a better fit because it spreads visible patches more evenly. However, simply high-pass filtering white noise does not explicitly control the separation between visible patches, so small clusters remain.
R-BN addresses this with a lightweight optimization. We draw $K$ candidate masks $\{M^i\}_{i=1}^{K}$ by thresholding noise at the target ratio $\gamma$. Then, for each spatial position $P=(x,y)$ in randomized order, we consider a local window $U_P^i \in \mathbb{R}^{\Delta\times\Delta}$ and count visible patches aligned with $P$ along four orientations: horizontal ($d_1^i$), vertical ($d_2^i$) and the two diagonals ($d_3^i, d_4^i$). This gives a clustering score
$$ S_P^i = w_1 d_1^i + w_2 d_2^i + w_3 d_3^i + w_4 d_4^i . $$
The least clustered candidate $\hat{\imath} = \arg\min_i S_P^i$ claims the patch as visible; every other candidate masks it. The loop runs until each mask holds $(1-\gamma)N$ visible patches.
Pseudocode: Regularized Blue Noise mask generation
