somehow Perennialism/Traditionalism seems to be associated to elitist and Fascist tendancies. Is that an accident? Are there thinkers out there who are serious thoughtful Perennialists but who are also progressive and democratic?
There are "progressive" perennialists, but I have not heard of many "progressive" Traditionalists. This is presumably because Guénon saw progress as an illusion, an important point with which few (if any) later Traditionalists have disagreed. This does not mean, however, that all Traditionalists have gone to the other extreme and become fascists. Many are not particularly interested in politics, progressive, fascist, or centrist.
There is an important political stream within Traditionalism, though, which follows Julius Evola and Alexander Dugin, and has often been described as "fascist," though I myself prefer a label such as "new right" or "far right." This has only limited implications for the spiritual Traditionalism represented by people such as S. H. Nasr, however. Since the 1920s, political Traditionalism has developed separately from other varieties of Traditionalism. Certainly, both spiritual and political Traditionalism ultimately derive from Guénon, even though Guénon himself had little interest in politics. But both S. H. Nasr and Sayyid Qutb ultimately derive from the Prophet Muhammad, without having anything much to do with each other.