It's tempting to say that if the Americans don't like him, he must be doing well, but there's a little more to it than a glib backhander.
In fact, he comes across as a particularly intelligent, thoughtful person, as I found when I listened to an interview on BBC radio recently.
The position is that of a diplomat, first and foremost – and a very difficult position it is. He has no real muscle, and all the while must defend the UN from US agendas and the tendency of the US to treat the UN like a handbag.
He's been vocal constantly on issues of significant peril, such as Darfur and HIV/AIDS. And in his sunset days, he gave a pointed and strong message on global warming to the most recent climate change conference, in Nairobi.
But he can only attempt to persuade, not enact. And he bore on his shoulders the weight of a relatively weak organisation that can only achieve what its members want to achieve.
It's interesting to note that a survey of US political cartoons on the UN was almost entirely negative. The only distraction for the cartoonists was the clownish departing US ambassador to the UN, John Bolton.
Still, even the US remembers at times that there is nobody else for the job but the UN. Whether America approves of the results or not, it's clear a UN presence reduces bloodshed and ameliorates human disasters. Not to mention the various agencies at work on such issues as refugees and human rights.
Convenient for all when they are needed, ignored when they are not. It's a brave job that Annan took on. Let's home the new Secretary-General (Korea's Ban Ki-moon) has the same wisdom, compassion, and diplomacy as Annan.