Actually, no. The size of the galaxy has no impact on the difficulty level. Instead, it just affects the quantity of resources available, as well as the duration of the game.
The difficulty level is determined from the number of major AI players, as well as their intellegence. It is a summation, meaning that 1 single "Incredable" AI is about equal to 5 "Brilliant" AIs. (Which is not exactly true, but close enough.)
The size of the galaxy is really a placebo to the sense of the difficutly level - smaller galaxies seem harder because of the lack of room, while large galaxies seem hard due to the quantity of enemy forces. Regardless of the size of the map in this genre of games, there is no impact on how hard the game will be.