Read the label of the soap. Sometimes just because it is labeled 'organic' doesn't necessarily make it so. Depending on the soap you may have a reaction to one of the ingredients as well. I would suggest using just a castile soap (without anything added to it) which is about as mild as you can get.
Also, make sure you rinse the soap really good off of your skin.