Fannie Mae survey echoes polls by the University of Michigan and the Conference Board that found uncertainty over tariffs is weighing on consumer confidence.