Background Aggressive breast cancers produce abnormal peritumoral stiff areas, which can differ between benign and malignant lesions and between different subtypes of breast cancer. Purpose To compare the tissue stiffness of the inner tumor, tumor border, and peritumoral stroma (PS) between benign and malignant breast masses by shear wave elastography (SWE). Material and Methods We enrolled 133 consecutive patients who underwent preoperative SWE. Using OsiriX commercial software, we generated multiple 2-mm regions of interest (ROIs) in a linear arrangement on the inner tumor, tumor border, and PS. We obtained the mean elasticity value (Emean) of each ROI, and compared the Emean between benign and malignant tumors. Odds ratios (ORs) for prediction of malignancy were calculated. Subgroup analyses were performed among tumor subtypes. Results There were 85 malignant and 48 benign masses. The Emean of the tumor border and PS were significantly different between benign and malignant masses ( P < 0.05 for all). ORs for malignancy were 1.06, 1.08, 1.05, and 1.04 for stiffness of the tumor border, proximal PS, middle PS, and distal PS, respectively ( P < 0.05 for all). Malignant masses with a stiff rim were significantly larger than malignant masses without a stiff rim, and were more commonly associated with the luminal B and triple negative subtypes. Conclusion Stiffness of the tumor border and PS obtained by SWE were significantly different between benign and malignant masses. Malignant masses with a stiff rim were larger in size and associated with more aggressive pathologic subtypes.