Currently, it falls back to trying every key available in keyholder, which will error unless it succeeds before hitting MaxAuthTries (default of 6).
From discussion in #wikimedia-releng, while debugging T313259:
<hasharAway> so maybe on the deployment server manually amend /srv/deployment/phabricator/deployment/scap/scap.cfg <hasharAway> and add <hasharAway> `keyholder_key: phabricator` * brennen tries that <hasharAway> $ ls -la /etc/keyholder.d/phabricator <hasharAway> -r--r----- 1 root keyholder 1766 Nov 30 2020 /etc/keyholder.d/phabricator <hasharAway> I don't know why it would have broken <hasharAway> maybe due to some codechange done recently in scap <brennen> seems like it might be working <hasharAway> I might be the one to blame <hasharAway> cause I know close to nothing about scap code and if I know about that get_keyholder_key method it must be that I have altered it recently <brennen> that did it - thanks hasharAway! <hasharAway> well <hasharAway> great :] <hasharAway> Using key: /etc/keyholder.d/phabricator <hasharAway> from the scap log <hasharAway> while previously we had: <hasharAway> `Running remote deploy cmd ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'phabricator/deployment', '-g', 'default', 'fetch', '--refresh-config']` <hasharAway> `Unable to find keyholder key for phab_deploy` <hasharAway> `['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'phabricator/deployment', '-g', 'default', 'fetch', '--refresh-config'] (ran as phab-deploy@phab2001.codfw.wmnet) ` <hasharAway> if we can't find a keyholder key using the ssh_user or the keyholder_key config value if it is set <hasharAway> then I think scap should abort entirely <hasharAway> else it tries to do every single keys from the keyholder ( see above logstash link) <brennen> right, and then just fails on too many auth attempts <hasharAway> and fails unless you deploy with one of the first 6 keys <brennen> my guess is that at one time the fallback might have worked because there weren't very many keys to try