Yes, for a small number of people that might be a factor (and for one of my friends it actually was one of the reasons he voted Leave) but for far more people it was "bloody immigrants", "muslims" and other small minded shite.
That might seem like small minded shite but for people in areas where they feel that they are being crowded out by a foreign culture, and often one that is actively hostile to UK culture it isn't. When you get immigration at levels where people of a common foreign culture can form large groups of a different culture you lose the ability to integrate them. Ghettos become self-sustaining. And as you get more immigrants you get a fracturing of the social cohesion - divide and conquer - very useful to the ruling classes and part of the reason for unfettered immigration.
There is good reason to be suspicious of ghettoed cultures, too. When they don't need to integrate and if they naturally see themselves as being more righteous than the host population, animosities will be taught and philosophies completely antithetical to western tolerance fester. Those are real concerns that politicians seem indifferent to but they aren't the only concerns.
When you get large integrated states you get, naturally, centralized control, and thence you get even greater concentrations of wealth and power. You also have a monocropping of governance which can have efficiencies but also increases brittleness.
When every country becomes effectively a single centrally run country then the citizens don't have a choice of leaving - which is something that chastens states - when a state is too poorly run the people flee. But without this disciplining constraint the mega-state has no downside in the long term to recreate a fuedal state. The Soviet Union basically had to build walls and heavily control passports to manage flight. The EU just grows to encompass all available land so there is no-where to go. Part of the dream of Europe is that you will be able to go from any country to any other and everything will be the same, but that's also the great threat. Borders are good, local governance is good - and hopefully the people will now start to take on the job of governing which they had abdicated to the technocrats.