The Red Light District is a really small section of the most touristic part in the centre of Amsterdam, it's easy to avoid if you don't want to be confronted with that kind of thing.
As far as drugs go, you won't see people using drugs on the street, except for tourists who feel obligated to smoke weed as soon as they set foot in this great city.
That being said, Amsterdam has one of the most lively club scenes in the world and people do tend to use drugs when they go to parties, but it's pretty much the same in other big cities and never bothered me to be honest. People are really friendly and worst case scenario they will ask you if you're having as much fun as they are, so nothing to worry about.