The Handkerchief Code in its modern form is said to have originated in New York in the early '70s, where gay and bi-sexual men found it a novel way to show potential partners exactly what they were into. These were post-Stonewall pre-AIDS days, where hedonism in the LGBT community in the USA was paramount, and cruising was still a large part of the gay lifestyle.
Different coloured hankies represented different sexual practises. In fact the colours themselves often gave some logical indication of the practise it signified, ie: yellow for water sports, brown for scat and so on.
If the hanky is worn on the left hand side, the wearer is considered to be the giver or 'top' in the action. On the right hand side means the wearer is on the receiving end, or 'bottom'. Pretty simple stuff really.
Regional variations do apply, but there are some generally accepted standards.
Hustler / Prostitute
This handy guide, provided by San Francisco store 'Image Leather', goes into more detail.
Usage of the code fell into decline after the AIDS epidemic devastated the communities and lifestyle associated with it, but many shops catering to the S&M market still carry coloured hankies.