Tearing the garment of the Kohein Gadol, even at the opening, is not only destructive, it’s disgraceful. So, not only are we forbidden to tear his special working clothes, the me’il (his robe) had to be reinforced around the neck to prevent even accidental damage. (“Tearing” it includes cutting, as with scissors. Both actions are equally prohibited.)
Again, the reason is that these uniforms are meant to instill proper awe, which they would fail to do if they were in a state of disrepair. Clothes are taken pretty seriously in Judaism and not just in terms of tzniyus (modesty). For example, the Talmud speaks harshly of scholars who go out in public wearing stained clothes, as doing so is a disgrace to their station (Shabbos 114a).
The prohibition against tearing the robe of the Kohein Gadol applies to both men and women when the Temple is standing. In the Talmud, it is discussed in tractate Yoma on page 72a. It is codified in the Mishneh Torah in the ninth chapter of Hilchos Klei HaMikdash and it is #88 of the 365 negative mitzvos in the Rambam’s Sefer HaMitzvos.