Over the period 2005 to 2020, Black borrowers with mortgages insured by Fannie Mae or Freddie Mac paid interest rates that were almost 50 basis points higher than those paid by non-Hispanic white borrowers. We show that the main reason is that non-Hispanic white borrowers are much more likely to exploit periods of falling interest rates by refinancing their mortgages or moving. Black and Hispanic white borrowers face challenges refinancing because, on average, they have lower credit scores, equity, and income. But even holding those factors constant, Black and Hispanic white borrowers refinance less, suggesting that other social factors are at play. Because they are more likely to exploit lower interest rates, white borrowers benefit more from monetary expansions. Policies that reduce barriers to refinancing for minority borrowers and alternative mortgage contract designs that more directly pass through interest rate declines to borrowers can reduce racial mortgage pricing inequality.