TL;DR
Starting from OpenSSH 9.0/9.0p1 released on 2022-04-08, scp command switches from using the legacy SCP protocol to using the SFTP protocol by default, and Dropbear doesn’t (fully) support SFTP protocol.
Add -O option to scp command to use legacy SCP protocol when interacting with Dropbear.
Long version
scp command is quite a convinent way to copy files over SSH and it’s a must-have utility in my daily work to deploy files to the embedded system at runtime.
However, after a rolling upgrade some time in Arch Linux, scp command stopped working and kept complainting that it couldn’t find sftp-server:
$ scp -i ~/.ssh/root_id_rsa fitimage [email protected]:/root
sh: /usr/lib/openssh/sftp-server: No such file or directory
scp: Connection closed
Where’s sftp-server
The first check was of cause to see if sftp-server exists in /usr/lib/openssh/ and if it doesn’t where it is then.
locate sftp-server showed that the actual location of sftp-server is /usr/lib/ssh/sftp-server instead (Note: it’s ssh not openssh).
$ locate sftp-server
...
/usr/lib/ssh/sftp-server
...
First attempt with -D
By searching around scp(1), I found -D option which can be used to specify SFTP server path:
-D sftp_server_path
When using the SFTP protocol support via -M, connect directly to a local SFTP server program rather than a remote one via ssh(1). This option may be useful in debugging the client and server.
However, when using -D option with the correct SFTP server path, the destination always replied with “Permission denied” even if the user (root here) has the required permission to write to that directory:
$ scp -D /usr/lib/ssh/sftp-server -i ~/.ssh/root_id_rsa fitimage [email protected]:/root
scp: dest open "/root/fitimage": Permission denied
scp: failed to upload file fitimage to /root
It works with -O
Going back to scp(1), -O (big-oh) option rang a bell to me:
-O
Use the legacy SCP protocol for file transfers instead of the SFTP protocol. Forcing the use of the SCP protocol may be necessary for servers that do not implement SFTP, for backwards-compatibility for particular filename wildcard patterns and for expanding paths with a ‘~’ prefix for older SFTP servers.
It turned out that big-oh is what I need!
$ scp -O -i ~/.ssh/root_id_rsa fitimage [email protected]:/root
fitimage 100% 29MB 4.2MB/s 00:06
And from the description of -O option, I recall that filename wildcard and ~ prefix didn’t work before and now I know how to make it work.
Why SCP protocol but not SFTP protocol
The system I was working on runs dropbear as SSH server, which is common in embedded ecosystem.
According to Comparison of SSH servers on Wikipedia, Dropbear partially supports SFTP. And Dropbear’s changelog says it added support of repeated requests in a single session if SCP clients try SFTP then fall-back to SCP:
Allow repeated requests in a single session if previous ones fail - this fixes PuTTY and some other SCP clients, which try SFTP, then fall-back to SCP if it isn’t available. Thanks to Stirling Westrup for the report.
So it looks like Dropbear doesn’t support SFTP protocol (or at least not fully support), and scp command from OpenSSH on my PC uses SFTP protocol by default but doesn’t try SCP protocol as a fall-back unless you use -O option explicitly to use SCP protocol only.
Why did it work without -O before
OpenSSH Release Notes for 9.0/9.0p1 released on 2022-04-08 explains it all:
Potentially-incompatible changes
This release switches scp(1) from using the legacy scp/rcp protocol to using the SFTP protocol by default.
Legacy scp/rcp performs wildcard expansion of remote filenames (e.g. “scp host:* .”) through the remote shell. This has the side effect of requiring double quoting of shell meta-characters in file names included on scp(1) command-lines, otherwise they could be interpreted as shell commands on the remote side.
This creates one area of potential incompatibility: scp(1) when using the SFTP protocol no longer requires this finicky and brittle quoting, and attempts to use it may cause transfers to fail. We consider the removal of the need for double-quoting shell characters in file names to be a benefit and do not intend to introduce bug-compatibility for legacy scp/rcp in scp(1) when using the SFTP protocol.
Another area of potential incompatibility relates to the use of remote paths relative to other user’s home directories, for example - “scp host:~user/file /tmp”. The SFTP protocol has no native way to expand a ~user path. However, sftp-server(8) in OpenSSH 8.7 and later support a protocol extension “[email protected]” to support this.
In case of incompatibility, the scp(1) client may be instructed to use the legacy scp/rcp using the -O flag.
Have fun!