Having 3 or more identities often causes authentication to fail before it gets around to trying password authentication (or even all the possible keys). Recommend configuring the client to turn off PubkeyAuthentication by default (so that hosts that you don’t have a key for will prompt for a password) and specify which key to use on the appropriate hosts using IdentityFile (might need to specifically turn PubkeyAuthentication back on, I don’t remember how openssh handles having a default host block with specific host blocks)
That is indeed Don Quixote and I believe you’ll find that he is tilting at that windmill