Skin lesion segmentation supports computer-aided dermatology, but blurred boundaries, low contrast, and marked variation in dermoscopic appearance make reliable delineation difficult. Foundation models such as the Segment Anything Model (SAM) offer strong visual priors, although their sensitivity to prompt quality limits their reliability in few-shot medical settings. We present PADS, a novel perturbation-aware distillation framework for robust skin lesion segmentation under prompt variability. During training, we perturb bounding-box prompts derived from ground-truth masks to simulate realistic prompt noise. A lightweight relevance module then estimates channel-wise feature importance under these perturbations and guides selective distillation from MedSAM into a compact UNet. This training improves robustness to imperfect or noisy prompts, allowing pseudo-prompts to support reliable segmentation. At inference, the UNet either produces the segmentation directly or supplies a pseudo-prompt for optional MedSAM refinement. Experiments on ISIC 2018 and cross-dataset evaluation on PH2 show improved few-shot segmentation accuracy and slower performance degradation as prompt perturbation increases, compared with standard distillation. PADS adds only limited computational overhead relative to MedSAM. The experimental results support perturbation-aware distillation as a practical approach to robust few-shot medical image segmentation.