Why dark matter is more diffuse than ordinary matter

It appears that galaxies form around clumps of dark matter, so the large scale structure of the universe is determined by the distribution of dark matter, and ordinary matter is attracted to these clumps of dark matter to form the galaxies we see today. Given that it is dark matter that clumps together first, and only then is ordinary matter attracted to these clumps to form galaxies, why is it ordinary matter that ended up being clumped in much more dense objects such as stars or galaxies? It's been a question that has been plaguing me for a while, so I asked about it on Astronomy Cast.

The evidence so far seems to indicate that dark matter only weakly interacts with itself, with gravitation being the only force that has any major influence. On the other hand, ordinary matter experiences the electromagnetic force and the strong force, which overcomes gravity to a certain extent and prevents many dense objects from collapsing into black holes, such as in white dwarves and neutron stars. When it's dark matter that clumps first, only after which ordinary matter follows, what stops dark matter from clumping and collapsing all the way to black holes? What makes ordinary matter clump even further than dark matter when there's nothing stopping further clumping of dark matter?

Galaxy Cluster MACS J0025.4-1222
Galaxy Cluster MACS J0025.4-1222

The thing I was missing is that what keeps ordinary matter apart in objects such as white dwarves is also responsible for gathering ordinary matter together. When clumps of ordinary matter collide with each other, they slow down due to friction, which is basically the electromagnetic force. On the other hand, clumps of dark matter just pass through each other since they apparently don't experience anything like the electromagnetic force. This can be seen clearly in the Bullet Cluster and MACS J0025.4-1222, where clouds of ordinary matter collide and gather together, whereas clouds of dark matter just go on their merry way.

Friction is why ordinary matter continued to clump into stars and galaxies, while the lack of friction is the reason why dark matter is much more diffuse despite being the scaffolding around which ordinary matter gathered. Gravity caused dark matter to clump to a certain degree, but nothing makes the dark matter slow down so they can clump even further. Whereas clouds of ordinary matter attracted to clumps of dark matter start experiencing friction once they become dense enough, so the particles making up ordinary matter clouds slow down and clump further into galaxies. More friction clumps ordinary matter even more to form stars and planets.

If I'm right, then it would also answer something else I've been wondering about: How much of the supermassive black holes in the centers of galaxies formed out of ordinary matter compared to dark matter? Most of the mass should be due to ordinary matter, with only a miniscule portion being due to dark matter. It wouldn't be the case that supermassive black holes formed out of dark matter attracted ordinary matter into galaxies. Then again, I'm not an expert, nor have I seen any concrete references by professional astronomers, so I may be completely wrong about why dark matter is more diffuse than ordinary matter.

Is there anyone out there knowledgeable enough to tell me whether I have the right idea or whether I'm completely off the mark?