This paper proposes a hierarchical approach to design the sensing matrix of the SPC, such that the pixel clustering task can be performed directly using the compressed infrared SPC measurements without a previous reconstruction step. Specifically, a sensing matrix is designed to extract features directly from the compressed measurements at each hierarchy step. Then, a final segmentation map is obtained through majority voting in the partial clustering results.