Security Keys

[carloscabanero]

I want to open a discussion for what our options supporting Security Key are. Please read and chime in. I want to make sure we have enough support behind a solution before moving development ahead on this again. If we have that, we could possibly have something for October.

This month we took a deep look at what is the current situation and what the opportunities here are. Long story short, SSH WebAuthn is still a bet on the future and during this year, support has not improved. But the good news are that we may have a different option going forward.

Requirements

• The solution should work across all devices. Specially, iPad and USB-C.
• The solution should allow to use a key stored in the HW across all devices. Ie, moving a key from Desktop to your iPad or phone.
• The solution should play with other tools like the agent. Ie. Having a Git key in the Secure Key and using it between Desktop and mobile.
• Ideally, not restricted to one vendor, but will take it if it is the only option.
• Ideally, not depending on one vendor SDK, but will take it if it is the only option.
• Ideally, the solution should come from our side and not depend on third parties making changes.
• Not necessary to work between platforms (from macOS to Windows).

Supporting sk-ecdsa and sk-ed25519 on iOS / iPadOS

WebAuthn

This is our current approach. OpenSSH supports both U2F and WebAuthn. Although keys are identified the same, the signatures types different: Protocol info for [email protected] & [email protected]. This is because the content of the signed blob, particularly the clientData, differs on both. WebAuthn is backwards compatible, but you cannot have a U2F server deal with a WebAuthn signature.

Although in the “web world” WebAuthn has phased out U2F, in the OpenSSH world it looks like U2F has received most of the backing. Ie. GitHub has adopted passkeys for web but still does not support them for ssh log in. We could try to be more active on getting it enabled in more places, but we don’t have much leverage.

WebAuthn is very cool though and it works through all keys. Synchronizing between devices should be straightforward as well. One limitation was the SSH Agent but this could be fixed, at least in the Blink Agent.

U2F

Then I looked into supporting U2F signatures through the Yubikey SDK. Unfortunately, the U2F application in Yubikey does not work over USB-C in the iOS SDK. It does support it over NFC and MFI (Lightning). Libfido2 works using PC/SC, which is not seem possible on iOS. A similar thing would be the Yubikey using SmartCard over USB-C, but that interface does not have U2F available in the device. I think the HID FIDO interface (which would be the useful one), is the one they used for WebAuthn but we cannot tap into it. Will try to reach out to Yubico on this, but please forward the question if you know anyone there.

If we want to be more flexible with the requirements, we could make it iPhone only with NFC, MFI, etc. This is what Termius does, but I do not think it is worth the effort. It would only work on Yubikeys, we would have to embed their SDK, etc… To solve the iPhone-only issue, we could maybe route log in requests from our agent to a nearby phone. I think this would be already too complicated.

Supporting PKCS11 on Smart Cards

iOS16 added Smart Card integration. This gave me the idea that we could do something like PKCS11 on OpenSSH (suggested by Yubikey on PIV section).

This seems even supported out of the box on macOS. I tried it and it took me a while to set it up, but it worked. I had to download all the Yubikey CLI tools, then install ssh from Homebrew so other libraries could be enabled, etc… https://apple.stackexchange.com/questions/448697/how-can-i-use-my-yubikey-certificate-as-ssh-private-public-keys-on-macos

But people seems to be happy with it and even call it “easy”, “three-steps”: https://news.ycombinator.com/item?id=17683061. IMO, not as easy as it should be though and they have had their issues: Yubico/yubico-piv-tool#319. But it seems standard enough.

The good news is: I did a quick test and it would not be super hard to support. It seems like it would even check all the boxes in our requirements. And because in Blink we control the agent, it would actually be super easy and compatible with already installed keys, etc… It would work with ECDSA keys, ED25519, etc... It would easily work with certificates and they would actually be handled at the key level.

Hence the question, would this be the preferred solution? Is this the way everyone has been doing keys on the desktop all this time before sk-xxxx? Is this community and even company friendly (as it looks like)?

Leave your comments

Would be interesting to hear from people already doing this to figure out where each solution stands. Specially those who have more strict requirements (say certificates, etc...). Unfortunately there does not seem to be a one-size fits all and our options are even more reduced on iOS. I may have missed something as well, so please jump in.

[jac-cbi]

Yup. I've relied on Yubikey/PKCS11/gnupg for several years (ed25519). It was a huge PIA to set up, including getting the pin prompt to pop up in the right place. But once set up, it's solid.

Webauthn / FIDO2 promise to make the set up flow easier, but I still have trouble with use.

If blink could cleanly support the PKCS11 flows of key creation, use, and deletion, I'd just use that. Especially if I could do ed25519 :)

[carloscabanero]

Supporting the creation flow for self-signed keys (as usually these keys are certs), should be easy. It would actually be easier than downloading tools in your computer and figuring out to connect them.

If you need a key certified by someone else, then in that case I think we keep ourselves out of the way, as I assume someone else will have to do that for you. In this scenario we would just handle usage.

But please, let me know if you think I may be missing something

[DarkDimius]

FWIW, it might be useful to consider if you want to make this decision based on implementation constraints or the user cohorts it will support.

1. if you do it based on implementation constraints today, you correctly highlight there is not a single sdk/protocol and interop is poor as they have different key signatures. If we were to make decisions based on this, we'd probably come with a solution that is a future proof one but does not work today.
2. if you do it based on user cohorts, you might want to consider some statistics on what devices people have. I might be living a bubble, but around me, everybody uses yubikeys - although there was some competition with google titan keys, after security flaws a couple years ago, I haven't seen many people get one.

I also want to caution on building a strategy that only works in future. Security devices tend to be sticky and have rollouts that are hard to coordinate. PKCS11 is old, but is alive and well in many places. I'd be surprised out of organizations that adopted u2f-only keys more than 50% will migrate to WebAuthn based ones within next 5 years(again, because there is no good interop on either server or client side and thus doing a mixed setup is a pain).

[carloscabanero]

Thanks a lot for those insights. Although U2F only keys may become a thing in the future, I agree that this definitely won't happen overnight. It looked like PKCS11 was the way to go from what I had been reading, but wanted to confirm with those of you who may be using it or who have seen organizations implementing it, maybe even considered other options, etc...

One more question for you. Is the PGP Agent + Smart Card a better considered solution/combo for this scenario? I'm not sure we could support that one though, but curious.

[Tahpo2]

I have experience using WebAuthn/FIDO2 with ssh and using pgp keys via gpg. I have not used PIV, which I think is the thing Apple supports. I don’t understand why anyone would use U2F now that the FIDO2 standard is implemented in OpenSSH. I did not get the impression that it was very popular for anything other than website logins, although apparently GitHub uses it? I thought they updated to allow FIDO2/webauthn ssh keys?

FIDO2 is great, as long as you have a client and server that supports it. My Mac ssh client did not support it. I have yet to see an active Linux distro that won’t support it in with at most an extra call to the package manager for an sk module. This is what Blink currently supports via the iOS security key system. It works pretty well! By cleverly naming the key, I was able to get the key on my Yubikey created by Blink to be discoverable by OpenSSH. Now I can use my Yubikey to get onto my servers from any linux box that will take the ssh-keygen -K command, and I only have to delete one key from all of my servers if I ever lose my Yubikey. The one thing I would like to be able to do is use keys already on my Yubikey in a new install of Blink, which would make getting a new iPad easier for me. I assume that is technically infeasible due to the iOS security key implementation, although you can share passkeys via iCloud, so I could be wrong. I also assume that is not a common need among power users who have a good workflow for adding keys to a server.

I have also used my gpg auth key stored on my Yubikey for ssh authentication. This has the advantage that I think anyone with my public key can grant me access to a server without me having to send them a separate public key string. Also, I can quickly set up ssh access to servers from a new computer without moving public key strings around. The disadvantage is that it’s hard to set up. Once you know how, it only takes a minute, but you have to change environment variables and set up the gpg agent to interface with ssh. It’s a bad user experience, but it was my best option pre-2020 before anything supported FIDO2.

[carloscabanero]

I went down a really deep rabbit hole today, and although I will need a beer or two to get back from there, and I'm not fully sure what you need to do from your side, I can pin point that the issue is on the server side.

So the "Digest algorithm" is just a warning and not an error. The authentication process should continue after that.

I think you need to enable on the server side the [email protected] signature type. This shows in both my Ubuntu and macOS boxes as:

ssh_packet_ext_info: Received SSH_MSG_EXT_INFO
ssh_packet_ext_info: Follows 2 extensions
ssh_packet_ext_info: Extension: server-sig-algs=<ssh-ed25519,[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],ssh-dss,ssh-rsa,rsa-sha2-256,rsa-sha2-512>

The reason why you are able to connect with the extracted key, is because libfido2 will use U2F instead of a WebAuthn key, and your server probably support sk-ecdsa, but not webauthn-sk.

The full connection log sequence:

Starting connection to xxx.xx.xx.xxx
ssh_connect: current state : 6
ssh_packet_socket_callback: packet: read type 31 [len=188,padding=8,comp=179,payload=179]
ssh_packet_process: Dispatching handler for packet type 31
packet_send2: packet: wrote [type=21, len=12, padding_size=10, comp=1, payload=1]
crypt_set_algorithms2: Set output algorithm to [email protected]
crypt_set_algorithms2: Set HMAC output algorithm to aead-gcm
crypt_set_algorithms2: Set input algorithm to [email protected]
crypt_set_algorithms2: Set HMAC input algorithm to aead-gcm
ssh_init_rekey_state: Set rekey after 4294967296 blocks
ssh_init_rekey_state: Set rekey after 4294967296 blocks
ssh_packet_send_newkeys: SSH_MSG_NEWKEYS sent
ssh_packet_socket_callback: Processing 340 bytes left in socket buffer
ssh_packet_socket_callback: packet: read type 21 [len=12,padding=10,comp=1,payload=1]
ssh_packet_process: Dispatching handler for packet type 21
ssh_packet_newkeys: Received SSH_MSG_NEWKEYS
ssh_packet_newkeys: Signature verified and valid
ssh_client_connection_callback: session_state=6
ssh_packet_socket_callback: Processing 324 bytes left in socket buffer
ssh_packet_socket_callback: packet: read type 7 [len=304,padding=16,comp=287,payload=287]
ssh_packet_process: Dispatching handler for packet type 7
ssh_packet_ext_info: Received SSH_MSG_EXT_INFO
ssh_packet_ext_info: Follows 2 extensions
ssh_packet_ext_info: Extension: server-sig-algs=<ssh-ed25519,[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],ssh-dss,ssh-rsa,rsa-sha2-256,rsa-sha2-512>
Starting connection to xxx.xx.xx.xx
ssh_connect: current state : 7
Connection succeeded...
Authenticating...
Trying none...
packet_send2: packet: wrote [type=5, len=32, padding_size=14, comp=17, payload=17]
ssh_service_request: Sent SSH_MSG_SERVICE_REQUEST (service ssh-userauth)
ssh_packet_socket_callback: packet: read type 6 [len=32,padding=14,comp=17,payload=17]
ssh_packet_process: Dispatching handler for packet type 6
ssh_packet_service_accept: Received SSH_MSG_SERVICE_ACCEPT
packet_send2: packet: wrote [type=50, len=48, padding_size=12, comp=35, payload=35]
ssh_packet_socket_callback: packet: read type 51 [len=32,padding=16,comp=15,payload=15]
ssh_packet_process: Dispatching handler for packet type 51
ssh_packet_userauth_failure: Access denied for 'none'. Authentication that can continue: publickey
ssh_packet_userauth_failure: Access denied for 'none'. Authentication that can continue: publickey
Trying publickey...
ssh_agent_get_ident_count: Answer type: 12, expected answer: 12
ssh_agent_get_ident_count: Agent count: 1
ssh_userauth_agent: Trying identity ssh:yubikey
ssh_key_type_to_hash: Digest algorithm to be used with key type 15 is not defined
ssh_key_algorithm_allowed: Checking [email protected] with list <[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss>
packet_send2: packet: wrote [type=50, len=240, padding_size=13, comp=226, payload=226]
ssh_packet_socket_callback: packet: read type 60 [len=192,padding=5,comp=186,payload=186]
ssh_packet_process: Dispatching handler for packet type 60
ssh_userauth_agent: Public key of ssh:yubikey accepted by server
ssh_key_type_to_hash: Digest algorithm to be used with key type 15 is not defined
ssh_key_algorithm_allowed: Checking [email protected] with list <[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss>
packet_send2: packet: wrote [type=50, len=816, padding_size=12, comp=803, payload=803]
ssh_packet_socket_callback: packet: read type 52 [len=16,padding=14,comp=1,payload=1]
ssh_packet_process: Dispatching handler for packet type 52
ssh_packet_userauth_success: Authentication successful
ssh_packet_userauth_success: Enabling delayed compression OUT
ssh_packet_userauth_success: Enabling delayed compression IN
ssh_packet_need_rekey: packet: [data_rekey_needed=0, out_blocks=67, in_blocks=29
Connected to xxx.xxx.xx.xx

I would appreciate if you could send me a full log of the connection so I may be able to figure out if anything else may be going on. But I'm pretty sure it is just a configuration issue on the server.

[Tahpo2]

I turned up my logging verbosity and I’ll paste what I have below. I think you’re right that the “key type 15” error is not stopping anything and the problem is elsewhere.

Here is the tail end of my server logs when I try to connect with my sk key from Blink. I’ve replaced some things with [CAPITAL LETTERS IN BRACKETS] for privacy. There is no more log content for a few minutes after this before sshd throws a timeout message for the connection.

Mar 10 15:00:16 raspberrypi sshd[15387]: debug3: receive packet: type 50 [preauth]
Mar 10 15:00:16 raspberrypi sshd[15387]: debug1: userauth-request for user [USERNAME] service ssh-connection method publickey [preauth]
Mar 10 15:00:16 raspberrypi sshd[15387]: debug1: attempt 1 failures 0 [preauth]
Mar 10 15:00:16 raspberrypi sshd[15387]: debug2: input_userauth_request: try method publickey [preauth]
Mar 10 15:00:16 raspberrypi sshd[15387]: debug2: userauth_pubkey: valid user [USERNAME] querying public key [email protected] [KEY ID] [preauth]
Mar 10 15:00:16 raspberrypi sshd[15387]: debug1: userauth_pubkey: publickey test pkalg [email protected] pkblob ECDSA-SK SHA256:[KEY ID] [preauth]
Mar 10 15:00:16 raspberrypi sshd[15387]: debug3: mm_key_allowed: entering [preauth]
Mar 10 15:00:16 raspberrypi sshd[15387]: debug3: mm_request_send: entering, type 22 [preauth]
Mar 10 15:00:16 raspberrypi sshd[15387]: debug3: mm_key_allowed: waiting for MONITOR_ANS_KEYALLOWED [preauth]
Mar 10 15:00:16 raspberrypi sshd[15387]: debug3: mm_request_receive_expect: entering, type 23 [preauth]
Mar 10 15:00:16 raspberrypi sshd[15387]: debug3: mm_request_receive: entering [preauth]
Mar 10 15:00:16 raspberrypi sshd[15387]: debug3: mm_request_receive: entering
Mar 10 15:00:16 raspberrypi sshd[15387]: debug3: monitor_read: checking request 22
Mar 10 15:00:16 raspberrypi sshd[15387]: debug3: mm_answer_keyallowed: entering
Mar 10 15:00:16 raspberrypi sshd[15387]: debug1: temporarily_use_uid: 1000/1000 (e=0/0)
Mar 10 15:00:16 raspberrypi sshd[15387]: debug1: trying public key file /home/[USERNAME]/.ssh/authorized_keys
Mar 10 15:00:16 raspberrypi sshd[15387]: debug1: fd 5 clearing O_NONBLOCK
Mar 10 15:00:16 raspberrypi sshd[15387]: debug1: /home/[USERNAME]/.ssh/authorized_keys:1: matching key found: ECDSA-SK SHA256:[KEY ID]
Mar 10 15:00:16 raspberrypi sshd[15387]: debug1: /home/[USERNAME]/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
Mar 10 15:00:16 raspberrypi sshd[15387]: debug3: /home/[USERNAME]/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
Mar 10 15:00:16 raspberrypi sshd[15387]: Accepted key ECDSA-SK SHA256:[KEY ID] found at /home/[USERNAME]/.ssh/authorized_keys:1
Mar 10 15:00:16 raspberrypi sshd[15387]: debug2: auth_check_authkeys_file: /home/[USERNAME]/.ssh/authorized_keys: processed 1/2 lines
Mar 10 15:00:16 raspberrypi sshd[15387]: debug1: restore_uid: 0/0
Mar 10 15:00:16 raspberrypi sshd[15387]: debug3: mm_answer_keyallowed: publickey authentication test: ECDSA-SK key is allowed
Mar 10 15:00:16 raspberrypi sshd[15387]: debug3: mm_request_send: entering, type 23
Mar 10 15:00:16 raspberrypi sshd[15387]: debug3: send packet: type 60 [preauth]
Mar 10 15:00:16 raspberrypi sshd[15387]: debug2: userauth_pubkey: authenticated 0 pkalg [email protected] [preauth]
Mar 10 15:00:16 raspberrypi sshd[15387]: debug3: user_specific_delay: user specific delay 0.000ms [preauth]
Mar 10 15:00:16 raspberrypi sshd[15387]: debug3: ensure_minimum_time_since: elapsed 6.482ms, delaying 0.560ms (requested 7.042ms) [preauth]
Mar 10 15:00:16 raspberrypi sshd[15387]: Postponed publickey for [USERNAME] from [REMOTE IP]

Here is the tail end of the output from Blink with a -vvvv flag when I try that connection.

Trying publickey...
agent_talk: Request length: 1
ssh_agent_get_ident_count: Answer type: 12, expected answer: 12
ssh_agent_get_ident_count: Agent count: 1
ssh_userauth_agent: Trying identity [KEY NAME]
ssh_key_type_to_hash: Digest algorithm to be used with key type 15 is not defined
ssh_key_algorithm_allowed: Checking [email protected] with list <[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss>
packet_send2: packet: wrote [type=50, len=224, padding_size=4, comp=219, payload=219]
ssh_packet_socket_callback: packet: read type 60 [len=192,padding=12,comp=179,payload=179]
ssh_packet_process: Dispatching handler for packet type 60
ssh_packet_userauth_pk_ok: Received SSH_USERAUTH_PK_OK/INFO_REQUEST/GSSAPI_RESPONSE
ssh_packet_userauth_pk_ok: Assuming SSH_USERAUTH_PK_OK
ssh_userauth_agent: Public key of [KEY NAME] accepted by server
ssh_key_type_to_hash: Digest algorithm to be used with key type 15 is not defined
ssh_key_algorithm_allowed: Checking [email protected] with list <[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss>
agent_talk: Request length: 404

[Tahpo2]

In case it’s helpful, here are logs from a successful connection with the same key from a different client.

Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: receive packet: type 50 [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug1: userauth-request for user [USERNAME] service ssh-connection method publickey [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug1: attempt 2 failures 0 [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug2: input_userauth_request: try method publickey [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug2: userauth_pubkey: valid user [USERNAME] attempting public key [email protected] [KEY ID] [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: userauth_pubkey: publickey have [email protected] signature for ECDSA-SK SHA256:[KEY ID] [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_key_allowed: entering [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_request_send: entering, type 22 [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_key_allowed: waiting for MONITOR_ANS_KEYALLOWED [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_request_receive_expect: entering, type 23 [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_request_receive: entering [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_request_receive: entering
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: monitor_read: checking request 22
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_answer_keyallowed: entering
Mar 10 15:54:54 raspberrypi sshd[15635]: debug1: temporarily_use_uid: 1000/1000 (e=0/0)
Mar 10 15:54:54 raspberrypi sshd[15635]: debug1: trying public key file /home/[USERNAME]/.ssh/authorized_keys
Mar 10 15:54:54 raspberrypi sshd[15635]: debug1: fd 5 clearing O_NONBLOCK
Mar 10 15:54:54 raspberrypi sshd[15635]: debug1: /home/[USERNAME]/.ssh/authorized_keys:1: matching key found: ECDSA-SK SHA256:[KEY ID]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug1: /home/[USERNAME]/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: /home/[USERNAME]/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
Mar 10 15:54:54 raspberrypi sshd[15635]: Accepted key ECDSA-SK SHA256:[KEY ID] found at /home/[USERNAME]/.ssh/authorized_keys:1
Mar 10 15:54:54 raspberrypi sshd[15635]: debug2: auth_check_authkeys_file: /home/[USERNAME]/.ssh/authorized_keys: processed 1/2 lines
Mar 10 15:54:54 raspberrypi sshd[15635]: debug1: restore_uid: 0/0
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_answer_keyallowed: publickey authentication: ECDSA-SK key is allowed
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_request_send: entering, type 23
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_sshkey_verify: entering [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_request_send: entering, type 24 [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_sshkey_verify: waiting for MONITOR_ANS_KEYVERIFY [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_request_receive_expect: entering, type 25 [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_request_receive: entering [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_request_receive: entering
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: monitor_read: checking request 24
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_answer_keyverify: publickey ECDSA-SK signature using default verified
Mar 10 15:54:54 raspberrypi sshd[15635]: debug1: auth_activate_options: setting new authentication options
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_request_send: entering, type 25
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_request_receive_expect: entering, type 102
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_request_receive: entering
Mar 10 15:54:54 raspberrypi sshd[15635]: debug1: do_pam_account: called
Mar 10 15:54:54 raspberrypi sshd[15635]: debug2: do_pam_account: auth information in SSH_AUTH_INFO_0
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: PAM: do_pam_account pam_acct_mgmt = 0 (Success)
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_request_send: entering, type 103
Mar 10 15:54:54 raspberrypi sshd[15635]: Accepted publickey for [USERNAME] from [IP ADDRESS] port 51533 ssh2: ECDSA-SK SHA256:[KEY ID]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug1: monitor_child_preauth: user [USERNAME] authenticated by privileged process
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_get_keystate: Waiting for new keys
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_request_receive_expect: entering, type 26
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_request_receive: entering
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_get_keystate: GOT new keys
Mar 10 15:54:54 raspberrypi sshd[15635]: debug1: userauth_pubkey: sk_counter = 2993, sk_flags = 0x01 [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug1: auth_activate_options: setting new authentication options [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug2: userauth_pubkey: authenticated 1 pkalg [email protected] [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: user_specific_delay: user specific delay 0.000ms [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: ensure_minimum_time_since: elapsed 9.655ms, delaying 4.429ms (requested 7.042ms) [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_do_pam_account entering [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_request_send: entering, type 102 [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_request_receive_expect: entering, type 103 [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_request_receive: entering [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_do_pam_account returning 1 [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: send packet: type 52 [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug1: Enabling compression at level 6. [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_request_send: entering, type 26 [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: mm_send_keystate: Finished sending state [preauth]
Mar 10 15:54:54 raspberrypi sshd[15635]: debug1: monitor_read_log: child log fd closed
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: ssh_sandbox_parent_finish: finished
Mar 10 15:54:54 raspberrypi sshd[15635]: debug1: PAM: establishing credentials
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: PAM: opening session
Mar 10 15:54:54 raspberrypi sshd[15635]: debug2: do_pam_session: auth information in SSH_AUTH_INFO_0
Mar 10 15:54:54 raspberrypi sshd[15635]: pam_unix(sshd:session): session opened for user USERNAME by (uid=0)
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: PAM: sshpam_store_conv called with 1 messages
Mar 10 15:54:54 raspberrypi sshd[15635]: debug3: PAM: sshpam_store_conv called with 1 messages
Mar 10 15:54:54 raspberrypi sshd[15635]: pam_env(sshd:session): deprecated reading of user environment enabled
Mar 10 15:54:54 raspberrypi sshd[15635]: User child is on pid 15643

[carloscabanero]

Thanks a lot for the prompt extra information. Want to take a look while I still have everything fresh. Moving this to a separate issue, as we can exchange more info there.

[carloscabanero]

Ok, so created #1988 to continue the conversation there. Are you on TestFlight by any chance?

[sinoru]

Is it possible to interoperability with WebAuthn and U2F? I needs to more dig up.. It seems like CTAP1 and CTAP2 token can be converted each other: https://medium.com/webauthnworks/verifying-fido-u2f-attestations-in-fido2-f83fab80c355

[carloscabanero]

I looked into this back in the day and it isn't possible. The token is backwards compatible but not forward. Our signature is a WebAuthn signature (as generated by default on iOS), and it won't work if we tried to pass it as a U2F when the server validates it. But a U2F signature will work as a WebAuthn signature, as you found in that link.

[carloscabanero]

Went through a rabbit hole today and I want to document this a bit more. It would be possible to do this (not 100% sure), but not with the access to the APIs as we have them now. The problem is on the clientData that is being signed in U2F and in WebAuthn. We are able to pass the "challenge" to sign, but not hash of the clientData itself. The WebAuthn clientData is a JSON formed with other headers, and reconstructed server-side (reference on OpenSSH). If the authenticator is U2F only, the FIDO client can arrange things to pass the hash of WebAuthn to it that would be compatible. But we don't have such a low level access. And OpenSSH is just checking the "challenge" sent in case of U2F signature.

So no go...

[jamesez]

Everything works during key generation, but I can't connect to anything using the keys, even though they work fine from my Linux box. I get the "Digest algorithm to be used with key type 15 is not defined" error if I throw in a -v flag.

I am having the same issue - I was able to recover the key on my Mac, was able to ssh from macOS using libsk-libfido2.dylib, but from Blink on iPhone, I get this same digest not defined error. Server is running OpenSSH 9.5p1 (FreeBSD 14.0-release-p5).

FWIW: Prompt 3 is able to use the same sk key, but it requires NFC, which means it doesn't quite work on iPadOS.

[carloscabanero]

Yep, so the sk key will work with the Yubikey framework, and that means they won't work on iPads, which is our main use case scenario.

We honestly don't have any visibility on the internal process to use the security keys, as everything is handled by the OS. They work from my side after creating them on the iPad, but I'm not trying to then using them on the Mac. I wonder if that does something to the key that renders it non-accessible by the device afterwards?

I think the best path forward is to use PKCS#11, but this is a tricky one and I don't have the bandwidth to do this from my side. If anyone wants to help, we could collaborate and I would be happy to put a big bounty on it / use some of our funds to get this done.

[carloscabanero]

Hi! Please check out #1875 (reply in thread). Let's see if we can find what configuration may be missing so we can come up with proper docs for this.

[carloscabanero]

I finally got some time to test everything one more time, connecting to both macOS and Linux, and including the flows to extract the keys on both macOS and Linux and I can attest from my side that this is working.

HW: YubiKey 5C NFC and iPad Air (m1) and iPad Pro (m2). Still need to test on new iPhone.
macOS: Running OpenSSH_9.6p1, OpenSSL 3.2.1 30 Jan 2024 - I had to install from Homebrew as otherwise libfido2 wouldn't be straightforward (there is a full thread here, but atm, just brew install openssh did the trick.
Linux: Running Ubuntu 23.10, OpenSSH_9.3p1 (SSH-2.0-OpenSSH_9.3p1 Ubuntu-1ubuntu3.2). Out of the box.

So I'm honestly surprised because I thought at one point the whole test would fail. I had a very old key from my last tests that I totally forgot about, but I was still able to use it to connect to my local computer. Then I created a different key with the "ssh:" prefix so that, as mentioned before, OpenSSH will identify it and extract it.

I proceeded to extract the key on the Mac using ssh-keygen -K as mentioned by @Tahpo2:

• This was a bit tricky as out of the box is not supported. Once I had everything setup, I still had errors until I figured I needed a PIN for the FIDO2 application, otherwise the flow wouldn't go through. I did not set a passphrase. I checked the public keys extracted and the one from the iPad and confirmed they were the same.
• Once the key was extracted, I could log in using -i <key_name>. Please note that although the key is the same, the signatures used on both are totally different (U2F vs WebAuthn).
• I then went to the iPad and I was hoping that something would break after extracting the key. Like maybe something would flip on the Yubikey and invalidate it. The same PIN was now being requested on the iPad, so logging in is a whole dance of inserting, tapping, introducing pin, tapping again. But it worked!! And it is really cool 👍

On Linux. Same story, but working out of the box. All the machines are able to connect to each other using the same Yubikey generated by Blink.

So that's the state of things as of today.

[carloscabanero]

More info on implementation:

• WebAuthn is growing on me and looks like more services are enabling it and SSH this way is not out of the standard (https://github.com/w3c/webauthn/wiki/Explainer:-WebAuthn-Large-Blob-Extension). To get this out of beta, we would need to finalize and enable Agent, syncing, and support on File Provider.
• Looks like the best path for U2F signatures would be creating a driver for the Yubikey and then enable it in the Yubikey framework, as discussed FIDO2 support over USB-C Yubico/yubikit-ios#138.
• I now see PKCS11 as a great companion, so it would be: PKCS11 && (WebAuthn || FIDO2). Because Blink would manage the whole flow, it would actually be quite easy to setup on every device. I need a ~month full-time for this, and I won't have it for some time. Or someone who may want to jump in. Again, we are happy to hire someone if it is up to the task, so please reach out to us!