The Cost of Down-Scaling Language Models: Fact Recall Deteriorates before In-Context Learning