The World Cup takes place in Qatar from 20 November to 18 December
The England boss said the Gulf state workers "want the football to come to Qatar" because they "love" the sport.
Amnesty International said migrant labourers were more concerned with having their rights "fully protected".
Qatar, host of the World Cup finals from 20 November, has been strongly criticised for its human rights record.
Asked by news channel CNN how important football is given Qatar's human rights concerns, Southgate, 52, said: "In the end, the football is everything. It's why we're there, it's what we're there to do."
He added: "I've been out to Qatar several times and I've met with lots of the workers out there and they are united in certainly one thing, and that's that they want the tournament to happen.
"They want that because they love football. They want the football to come to Qatar. But we also need to be realistic about how much we're going to affect in a country that we don't control."
It is estimated up to 30,000 migrant labourers have been used on projects to build seven stadiums for the finals in Qatar, as well as a new airport, new metro and new roads.
The English Football Association (FA) has backed calls for compensation to be awarded for "any injury or death related to any construction project" for the World Cup.
The England squad also plans to invite migrant workers to the team's World Cup training base in Al Wakrah to engage with players.
"As an FA we've talked to human rights groups about what they would like to see, and we're trying to support those ideas with compensation for families who have lost workers and a worker's rights centre," Southgate told CNN.
"We're supporting the things we've been asked to support."
Amnesty International's migrants' labour rights researcher Ella Knight said: "Many workers in Qatar will of course be football fans, but what migrant workers have really stressed to us is the need to have their rights fully protected, to be paid properly, able to change jobs freely and to enjoy safe and dignified working conditions - before, during and after this tournament."
Knight added: "The World Cup is about to take place with vital labour reforms still very much unfinished business, and thousands of workers' abuses remaining unaddressed.
"The opening game is now less than three weeks away and Fifa is yet to commit to remedying workers and their families for the abuses they've suffered despite widespread support from the public, FAs, players and World Cup sponsors.
"We urge the FA to maintain its pressure on Fifa, pushing it to recognise and urgently address the suffering of workers without whom the World Cup simply wouldn't be possible."
Minky Worden, director of global initiatives at Human Rights Watch, said: "There are many migrant workers who are proud of the work they have done to build the World Cup in Qatar.
"But there are also many who have suffered preventable deaths and harms and until the deaths, loans, injuries and wage theft are compensated, it is not correct to say all migrant workers are 'united'.
"Any family who received a loved one and breadwinner home in a coffin without compensation from Fifa and Qatar cannot cheer the opening of this World Cup."
Southgate said previously that England's players cannot do much more beyond discussing human rights issues in the build-up to the World Cup.
England captain Harry Kane will wear a OneLove armband which seeks to promote diversity and inclusion and act as a message against discrimination.
Homosexual acts are illegal in Qatar because they are considered to be immoral under Islamic Sharia law, with punishments including fines, prison sentences of up to seven years - and even death.
In February 2021, the Guardian said 6,500 migrant workers from India, Pakistan, Nepal, Bangladesh and Sri Lanka had died in Qatar since it won its World Cup bid.
The number is based on figures provided by the countries' embassies in Qatar.
However, the Qatar government said the total was misleading, because not all the deaths recorded were of people working on World Cup-related projects.
The government said its accident records showed that between 2014 and 2020, there were 37 deaths among labourers at World Cup stadium construction sites, only three of which were "work related".
-- Courtesy of BBC Sport