A. This is a common problem. It can be difficult to know where to draw the line between polite attention and overbearing conduct, especially since people are so different. Some will be offended if you draw attention to their appearance instead of their professionalism, while others will be offended that they invested so much in a new hairstyle or garment to present a pleasant appearance to their coworkers yet no one even notices. It is always better to err on the side of caution, and we wrote a few weeks ago about the great severity Jewish tradition attaches to coarse language. But the dilemma remains and eventually almost everyone will find themselves at sometime in the situation you describe.
One aspect of apologies is the one you mention: to obtain forgiveness from someone we have offended. This is the aspect emphasized by Maimonides in the laws of repentance:
Transgressions against one's fellow man, such as one who hit his fellow or cursed his fellow or stole from him and so on, he is never forgiven [by God] until he gives his fellow what he is owed and [also] conciliates him. Even after he returned the money he owes him, he [still] needs to conciliate him and to ask forgiveness until he forgives him. Even if he only taunted him with words, he must conciliate him and entreat him until he forgives him. (1)
However, Rav Yaakov ben Asher ("Tur"), a very prominent early codifier, emphasizes a different aspect: simply having good relations between people. The Tur brings the exact same law as Maimonides does stating that one has to conciliate anyone one has sinned against. But whereas Maimonides describes it as part of the same process as repaying damagers, the Tur writes:
We do this in order that all of Israel should be content one with the other, and the Accuser will not find any place [discord] to accuse them. (2)
There are a number of subtle differences between the approaches. Perhaps a person really did nothing wrong, but someone misinterpreted or was misinformed. According to Maimonides' approach there would seemingly be no need to ask forgiveness because no atonement is needed if there was no sin. But according to the Tur it would still be appropriate to ask forgiveness or at any rate to explain the situation in order to maintain good relations.
There is yet another aspect of apology, which is to make a public acknowledgment of regret. Not only the victim of the slight but also the entire community can be apprised of the fact that no ill will persists between the parties. Thus, both Maimonides and the Tur state that if the person victimized is dead, one should bring ten people to his grave and make a public statement of asking forgiveness then.
It follows that you don't need to know that your co-worker took offense before you apologize. The best thing is to just state that you only meant to be polite and complimentary and you certainly didn't mean any offense, and that you are sorry if you caused any.
Especially if you are not certain the person took offense, it's best not to adopt too apologetic a tone. A person may then feel bad that they are oversensitive or that they caused you anguish. Be sincere but friendly. You didn't mean anything offensive and you don't need to be conscience stricken, but it is a good idea to go back to the person and be 100% certain that your coworker knows that you had and have no malice.
SOURCES: (1) Maimonides' Code, Laws of Repentance 2:9 (2) Tur Orach Chaim 606.