Compressive spectral imaging (CSI) acquires coded projections of a spectral scene reducing storage costs. The single-pixel camera architecture (SPC) excels among several CSI devices due to its low implementation cost. Traditionally, before applying any post-processing task, e.g., clustering, it is required to solve a computationally expensive optimization problem to reconstruct the 3D information. Instead, this paper proposes a hierarchical approach to design the sensing matrix of the SPC, such that the pixel clustering task can be performed directly using the compressed infrared SPC measurements without a previous reconstruction step. Specifically, a sensing matrix is designed to extract features directly from the compressed measurements at each hierarchy step. Then, a final segmentation map is obtained through majority voting in the partial clustering results. Through simulations and experimental proof-of-concept implementation, we demonstrate the efficient proposed alternative to estimate clustering maps without relying on oversampling sensing protocols.