Starting from OpenSSH 9.0/9.0p1 released on 2022-04-08,
scp command switches from using the legacy SCP protocol to using the SFTP protocol by default, and Dropbear doesn’t (fully) support SFTP protocol.
-O option to
scp command to use legacy SCP protocol when interacting with Dropbear.
scp command is quite a convinent way to copy files over SSH and it’s a must-have utility in my daily work to deploy files to the embedded system at runtime.
However, after a rolling upgrade some time in Arch Linux,
scp command stopped working and kept complainting that it couldn’t find
$ scp -i ~/.ssh/root_id_rsa fitimage [email protected]:/root sh: /usr/lib/openssh/sftp-server: No such file or directory scp: Connection closed
The first check was of cause to see if
sftp-server exists in
/usr/lib/openssh/ and if it doesn’t where it is then.
locate sftp-server showed that the actual location of
/usr/lib/ssh/sftp-server instead (Note: it’s
$ locate sftp-server ... /usr/lib/ssh/sftp-server ...
First attempt with
By searching around scp(1), I found
-D option which can be used to specify SFTP server path:
When using the SFTP protocol support via -M, connect directly to a local SFTP server program rather than a remote one via ssh(1). This option may be useful in debugging the client and server.
However, when using
-D option with the correct SFTP server path, the destination always replied with “Permission denied” even if the user (
root here) has the required permission to write to that directory:
$ scp -D /usr/lib/ssh/sftp-server -i ~/.ssh/root_id_rsa fitimage [email protected]:/root scp: dest open "/root/fitimage": Permission denied scp: failed to upload file fitimage to /root
It works with
Going back to scp(1),
-O (big-oh) option rang a bell to me:
Use the legacy SCP protocol for file transfers instead of the SFTP protocol. Forcing the use of the SCP protocol may be necessary for servers that do not implement SFTP, for backwards-compatibility for particular filename wildcard patterns and for expanding paths with a ‘~’ prefix for older SFTP servers.
It turned out that big-oh is what I need!
$ scp -O -i ~/.ssh/root_id_rsa fitimage [email protected]:/root fitimage 100% 29MB 4.2MB/s 00:06
And from the description of
-O option, I recall that filename wildcard and
~ prefix didn’t work before and now I know how to make it work.
Why SCP protocol but not SFTP protocol
The system I was working on runs dropbear as SSH server, which is common in embedded ecosystem.
According to Comparison of SSH servers on Wikipedia, Dropbear partially supports SFTP. And Dropbear’s changelog says it added support of repeated requests in a single session if SCP clients try SFTP then fall-back to SCP:
Allow repeated requests in a single session if previous ones fail - this fixes PuTTY and some other SCP clients, which try SFTP, then fall-back to SCP if it isn’t available. Thanks to Stirling Westrup for the report.
So it looks like Dropbear doesn’t support SFTP protocol (or at least not fully support), and
scp command from OpenSSH on my PC uses SFTP protocol by default but doesn’t try SCP protocol as a fall-back unless you use
-O option explicitly to use SCP protocol only.
Why did it work without
OpenSSH Release Notes for 9.0/9.0p1 released on 2022-04-08 explains it all:
This release switches scp(1) from using the legacy scp/rcp protocol to using the SFTP protocol by default.
Legacy scp/rcp performs wildcard expansion of remote filenames (e.g. “scp host:* .”) through the remote shell. This has the side effect of requiring double quoting of shell meta-characters in file names included on scp(1) command-lines, otherwise they could be interpreted as shell commands on the remote side.
This creates one area of potential incompatibility: scp(1) when using the SFTP protocol no longer requires this finicky and brittle quoting, and attempts to use it may cause transfers to fail. We consider the removal of the need for double-quoting shell characters in file names to be a benefit and do not intend to introduce bug-compatibility for legacy scp/rcp in scp(1) when using the SFTP protocol.
Another area of potential incompatibility relates to the use of remote paths relative to other user’s home directories, for example - “scp host:~user/file /tmp”. The SFTP protocol has no native way to expand a ~user path. However, sftp-server(8) in OpenSSH 8.7 and later support a protocol extension “[email protected]” to support this.
In case of incompatibility, the scp(1) client may be instructed to use the legacy scp/rcp using the -O flag.