I had permanent ink marking pens on hand for fabric and used them. I think the price for colored ink/paint pens is cheaper in Fabric, Craft or Hobby Stores. Youre looking for a store where women get the stuff to make the cutsie type scrapbooks that are all the rage right now. My LDS had specific pens that may work better and be easier for you to find but looked like the same thing to me.
I have a tag in a pocket on my BC (serial number) and used a permanent Sharpie. I wanted to mark my black integrated weight pouches for boat diving since I noticed there was a lot of similarity and used a Pen-touch 2mm tip opaque white. I needed to make the mark thick, not just a quick initial like on a document. Its pretty crude looking on the heavy cordura.
I also tried another kind on my wetsuit that went on real pretty, a pale blue with silver outline. It is no where near as obtrusive as the white, pretty hard to see.
I used different brands of markers labeled as permanent and waterproof (Sharpie, Pilot) on different solid items with equal results.