You lose points for receiving

negative reputation and playing in the arcade. I cannot remember if you lose points for having a post deleted

(losing whatever points you gained) or if it stays, but they could all be reasons for you losing points.