Make sure that your SSH config file (sshd_config) contains these settings:
sshd_config
PubkeyAuthentication yes PasswordAuthentication no
If you changed the config, make sure to restart sshd to activate ist:
service ssh restart