South Sudan's embassy and the Africa Inland Church arranged the journey to Malakal, in South Sudan's Upper Nile state, after Sudanese authorities asked them to leave their camp around the Shajara train station in southern Khartoum, said Kau Nak, the embassy's deputy head of mission.
"Some of them are still loading" and the last of the vehicles will leave by Wednesday, he told AFP.
The UN says an estimated 40,000 South Sudanese are living in the open at camps like Shajara in the Khartoum area. Many have been there for more than two years, since before South Sudan became independent in July 2011.
"These points have basically become squatter camps and the people are living in squalor," Mark Cutts, head of the United Nations Office for the Coordination of Humanitarian Affairs (OCHA) in Sudan, said on the OCHA website.
"All of them are waiting for means of transport. All of them lost their jobs. All of them lost their nationalities, and nobody's supporting them," said Nak.
Belongings were strapped to the roofs of the 21 buses which left on Tuesday with 16 trucks.
Nak said some of the travellers felt they will need their mattresses, chairs, cooking gear and other vital belongings to start a new life in the South.
"Some of them were born here," and worry they will have nothing in South Sudan, he said.
"These people sold their properties and are living in poor conditions with little or no access to even the most basic of services. People are constantly threatened with eviction as the landowners want their land back and residents complain about the squatter areas," OCHA said.
AFP last year visited one camp, Dar es Salaam, which resembled a junk yard wrapped in hessian. Residents tied cloth bags around metal crates, beds and other possessions to form crude shelters.
They had no running water and not even a toilet in their dusty outpost.
The International Organisation for Migration (IOM) had assisted thousands of South Sudanese to move south by river barge, train and emergency airlift by mid-2012.
But increased tensions between Sudan and South Sudan, along with a lack of funds, stopped most of the returns, aid workers said.
IOM has been working with the Africa Inland Church to transport small groups "but the governments of Sudan and South Sudan need to organise and fund larger scale returns," to ensure the migrants' safety, OCHA said.
"There's no kind of a major repatriation process that is going to take place," Nak told AFP, urging another church or non-governmental group to step in.
During an interview at his office last week, Nak showed a hand-written list of 267 names sent to him by people who heard about the church operation and asked to be repatriated as well.
At last count there were more than 100,000 Southerners in the Khartoum area alone, he said.
Other estimates place the total number of southerners in Sudan at between 220,000 and 350,000.
"Without the resumption of organised return movements to South Sudan, the lack of prospects for the majority of those wanting to return back home in safety and dignity is very troubling," the UN Refugee agency's deputy Sudan representative, Francois Reybet-Degat, said on the OCHA website early this month before tensions appeared to ease between Sudan and South Sudan.
The two countries agreed on a series of timetables to implement key economic and security agreements, including a re-opening of 10 border crossings.
If those pacts take effect then the stranded South Sudanese should have more options to get help in moving south, or choosing a life in the north, Nak said.
The small convoy on Tuesday will also give them hope, he said.