Check to make sure that your hanger isn’t bent like this picture.
Photo courtesy of Park Tool
If it is, have it bent back by your LBS. They have a special tool that will make that straight as an arrow. You could try bending it back by hand, but you run the risk of messing up the actual derailleur as well as breaking off the hanger.
