The term “adultery” specifically refers to a married woman. A married man and a single girl, while still impermissible, is not an adulterous relationship. This is because, Biblically-speaking, a man is allowed to have more than one wife. (Rabbinically this is prohibited and it was never considered a good idea. Whenever a man in the Bible had more than one wife – from Jacob to Elkanah – trouble was sure to follow. In fact, the relationship between two women with the same husband is called “tzaros” in Hebrew, from the root meaning trouble because they caused each other tzuris.)
Why should a man be permitted more than one wife but a woman be limited to one husband? This actually makes far more sense than one might think. First of all, if a man has two or three wives, all the kids know who both parents are. If a woman had more than one husband, paternity would be cast into doubt. That would affect many other things, from inheritance to who’s permitted to marry whom (maybe she’s your half-sister and maybe she isn’t – it all depends on who your father is!). Also, men went to war and tens of thousands might perish in battle. If men were not permitted to have multiple wives, many women would remain unmarried because there simply weren’t enough husbands to go around. So the idea that a married man could have more than one partner made sense. (Again, this is not a license for extramarital relations, just an explanation as to why relations with a married man do not fit the halachic definition of ni’uf, adultery.)
And so, having relations with another man’s wife breaks down the fabric of society not only by creating the questionable matters of paternity mentioned above, but also by violating the very-literally sacred bond of marriage (“kiddushin” in Hebrew, from the word “kodesh,” holy).
Jewish marriage takes place in two stages. The first, called erusin, is usually rendered “betrothal” in English. This is more than being engaged and, while the couple is not yet permitted to live together, they are considered married for the purposes of this law. (The penalties are different, though. For adultery with a betrothed woman, the penalty is stoning, while for a fully-married woman, it is strangulation.) Betrothal used to occur months before the final marriage, but nowadays we perform both parts together.
A woman who has had an adulterous relationship becomes forbidden to her husband. Not only that, she becomes forbidden to her adulterous lover! So she can’t divorce her husband and marry the adulterer. (See Talmud Sotah 26b.)
The prohibition against adultery applies to everybody in the world, Jewish and non-Jewish. This is #347 of the 365 negative mitzvos in the Rambam’s Sefer HaMitzvos and #124 of the 194 negative mitzvos that can be fulfilled today in the Chofetz Chaim’s Sefer HaMitzvos HaKatzar. See Talmud Sanhedrin 51b and 84b, et al. It can be found in the Shulchan Aruch starting in Even Ha’Ezer 17.