Some years ago, on a visit to Durban, a woman at a roadside market spoke to me in Zulu. I had romanticised ideas of Shaka Zulu's brave battles in my head and was pleased to be mistaken for Zulu but I had to explain why I could not respond. "I'm Nigerian," I said. Her face fell. And the possibility of friendship, it seemed to me, disappeared. I would learn later that Nigerians were generally thought of as fraudsters, drug dealers and cheats, that Nigerian men were "taking away" South African women, that Nigerian immigrants were "worse" than Zimbabweans and Malawis and Mozambiquens.

But South Africa continues to draw Nigerians. Because it is not as recognisably "African" in its infrastructure and opportunities, it ranks much higher than the rest of sub-Saharan Africa on the scale of cool. Wealthy Nigerians own property there. Companies shoot commercials there. Young men in my hometown dream of going to find jobs in what they call "SA". South Africa represents a kind of "doable" ambition, steps below Europe and the US, but still the sort of place that impresses your village relatives when you come back home at Christmas.

There are obviously Nigerians involved in drugs and fraud in South Africa, just as there are many more doing honest work, but for the estimated 1.5 million Nigerians who now live in South Africa, to be Nigerian is to walk under a cloud of negative stereotypes. The reasons are economic and historical. A system as viciously brutal as apartheid will, it seems to me, take at least as long to undo as it lasted.

South Africans and Nigerians (and indeed other African immigrant groups) have simply not had the time or the neutral space to grow an organic understanding of each other. The Nigerians arrive with their different, more distant colonial experience, with their mercantile spirit, with none of the conditioning of the South African menial wage-earning experience and – yes – with that swagger. They arrive in a vulnerable country where the legacy of institutional exclusion still thrives. They create spaces for themselves in whatever way they can and, of course, they arouse resentment.

And these are people who, like me, grew up in a Nigeria that was fiercely anti-apartheid. We all sang Free Mandela. In primary school, we collected money to free the brothers in South Africa. Perhaps this is the reason I found South Africa a disconcerting place to visit, in the end. I felt incapable of truly understanding it, ill-equipped to grasp meaning and nuance, in a way that I have not experienced anywhere else in sub-Saharan Africa. It cracked my pan-African idealism.

Chimamanda Ngozi Adichie is the author of Purple Hibiscus, which won the Commonwealth Writers' Prize for best first book and Half of a Yellow Sun, which won the Orange Broadband prize for fiction. She has also published numerous short stories, essays and poems. She divides her time between Nigeria and the US, where she is pursuing graduate work in the African Studies program at Yale University