Using SCP with ssh2 in Node.js Applications without SFTP

In today’s world, secure file transfer between servers is a critically important task. Typically, SFTP or SCP protocols are used for this purpose. While SFTP offers advanced features and is considered more modern, its use may be prohibited in some systems due to security or compatibility reasons. In such cases, the old but time-tested SCP protocol comes to the rescue.

Analyzing existing solutions for working with the ssh2 package in Node.js, it becomes apparent that many of them use the SFTP subsystem to implement SCP. However, SCP and SFTP are two different protocols. Unlike SFTP, SCP does not support interactive mode and cannot process command scripts, which means all commands must be passed directly through the command line.

Moreover, even though both SCP and SFTP use the same SSH encryption for file transfer with similar overhead, SCP usually operates significantly faster when transferring files, especially in high-latency networks. This is because SCP implements a more efficient transfer algorithm that does not require waiting for packet acknowledgments. This approach increases transfer speed but does not allow interrupting the process without terminating the entire session, unlike SFTP.

In this article, we will explore how to set up and use SCP with the ssh2 package in your Node.js applications. This is particularly relevant in situations where SFTP is unavailable, but secure file transfer via the SCP protocol is required.

How SCP Works and Practical Examples

The SCP protocol is used for secure file transfer between local and remote hosts over SSH. Unlike SFTP, SCP does not support interactive commands and operates on the principle of message exchange in a strictly defined order. Understanding this sequence is crucial when implementing SCP using the ssh2 package in Node.js.

When transferring a file from a remote host, the client and server exchange messages as follows:

1) Establishing Connection and Authorization: First, we establish an SSH connection with the remote server and perform authorization.

const connection = new Client();
...
connection.connect(connectionOptions);

2) Sending the SCP Command: Upon receiving the ready event, the client sends the scp command with the necessary options to the server via the exec channel.

connection.exec(`scp -f ${remoteFile}`, (err, readStream) => {...});

3) Initializing the Transfer: To prompt SCP to send a response, you must first send the initial readiness signal 0x00 to synchronize actions between the client and server.

const { Client } = require('ssh2');
const fs = require('fs');

const remoteFile = '/opt/test.tar.gz';
const connection = new Client();

connection.on('ready', () => {
  connection.exec(`scp -f ${remoteFile}`, (err, readStream) => {
    if (err) {
      console.error(err);
      connection.end();
      return;
    }
    // Send the initial acknowledgment byte
    readStream.write(Buffer.from([0]));
  });
}).connect({
  host: 'your_remote_host',
  port: 22,
  username: 'your_username',
  privateKey: fs.readFileSync('/path/to/your/private/key')
});

After sending the initial 0x00 byte, the server will start transmitting the file’s metadata and its contents. We need to process this data and save the file on the local machine.

Let’s add event handlers for readStream to properly process incoming data and save the file.

const { Client } = require('ssh2');
const fs = require('fs');
const path = require('path');

const remoteFile = '/opt/test.tar.gz';
const localFile = path.basename(remoteFile); // Local filename for saving
const connection = new Client();

connection.on('ready', () => {  
  connection.exec(`scp -f ${remoteFile}`, (err, stream) => {  
    if (err) {  
      console.error('Error executing SCP command:', err);  
      connection.end();  
      return;  
    }  

    let fileStream;  
    let fileSize = 0;  
    let receivedBytes = 0;  
    let expect = 'response'; // Current expected state  
    let buffer = Buffer.alloc(0);  

    // Function to send acknowledgment  
    const sendByte = (byte) => {  
      stream.write(Buffer.from([byte]));  
    };  

    // Send the initial acknowledgment byte  
    sendByte(0);  

    stream.on('data', (data) => {  
      buffer = Buffer.concat([buffer, data]);  
      while (true) {  
        if (buffer.length < 1) break;  
        if (expect === 'response') {  
          const response = buffer[0];  
          if (response === 0x43) {  
            expect = 'metadata';  
          } else if (response === 0x01 || response === 0x02) {  
            expect = 'error';  
            buffer = buffer.slice(1);  
          } else {  
            console.error('Unknown server response:', response.toString(16));  
            connection.end();  
            return;  
          }  
        } else if (expect === 'error') {  
          const nlIndex = buffer.indexOf(0x0A); // '\n'  
          if (nlIndex === -1) break; // Waiting for more data  

          const errorMsg = buffer.slice(0, nlIndex).toString();  
          console.error(`SCP Error: ${errorMsg}`);  
          buffer = buffer.slice(nlIndex + 1);  
          connection.end();  
          return;  
        } else if (expect === 'metadata') {  
          const nlIndex = buffer.indexOf(0x0A); // '\n'  
          if (nlIndex === -1) break; // Waiting for more data  

          const metadata = buffer.slice(0, nlIndex).toString();  
          buffer = buffer.slice(nlIndex + 1);  

          if (metadata.startsWith('C')) {  
            const parts = metadata.split(' ');  
            if (parts.length < 3) {  
              console.error('Error: Invalid metadata:', metadata);  
              connection.end();  
              return;  
            }  

            fileSize = parseInt(parts[1], 10);  
            if (isNaN(fileSize)) {  
              console.error('Error: Invalid file size in metadata:', metadata);  
              connection.end();  
              return;  
            }  

            // Create a write stream for the file  
            fileStream = fs.createWriteStream(localFile);  
            fileStream.on('error', (fileErr) => {  
              console.error('Error writing file:', fileErr);  
              connection.end();  
            });  

            // Send acknowledgment  
            sendByte(0);  

            expect = 'data';  
            receivedBytes = 0;  
            console.log('Starting file transfer...');  
          } else {  
            console.error('Error: Expected metadata line, received:', metadata);  
            connection.end();  
            return;  
          }  
        } else if (expect === 'data') {  
          if (receivedBytes < fileSize) {  
            const remainingBytes = fileSize - receivedBytes;  
            const bytesToRead = Math.min(buffer.length, remainingBytes);  
            fileStream.write(buffer.slice(0, bytesToRead));  
            receivedBytes += bytesToRead;  
            buffer = buffer.slice(bytesToRead);  
            if (receivedBytes === fileSize) {  
              expect = 'data_response';  
            }  
          } else {  
            expect = 'data_response';  
          }  
        } else if (expect === 'data_response') {  
          console.log('Bytes received:', receivedBytes);  
          if (buffer.length < 1) break; // Waiting for more data  

          const response = buffer[0];  
          buffer = buffer.slice(1);  

          if (response === 0) {  
            fileStream.end(() => {  
              console.log(`File ${localFile} saved successfully.`);  
            });  
            expect = 'end';  
            sendByte(0); // Send acknowledgment to finish  
          } else if (response === 1 || response === 2) {  
            expect = 'error';  
          } else {  
            console.error('Unknown server response after data transfer:', response);  
            connection.end();  
            return;  
          }  
        } else if (expect === 'end') {  
          // Transfer completed  
          connection.end();  
          return;  
        }  
      }  
    });  

    stream.on('close', () => {  
      console.log('Transfer completed.');  
      connection.end();  
    });  

    stream.stderr.on('data', (data) => {  
      console.error(`STDERR: ${data.toString()}`);  
    });  
  });  
}).connect({
  host: 'your_remote_host',
  port: 22,
  username: 'your_username',
  privateKey: fs.readFileSync('/path/to/your/private/key')
});

Step-by-Step Guide

In this section, we will take a detailed look at the example from the post, describing each step of the process of retrieving a file from a remote server using the SCP protocol in Node.js.

Step 1: Import Required Modules and Set Up Variables

const { Client } = require('ssh2');
const fs = require('fs');
const path = require('path');

const remoteFile = '/opt/test.tar.gz';
const localFile = path.basename(remoteFile); // Name of the file to save locally
const connection = new Client();

• Import Modules:
  • ssh2 – to establish an SSH connection and execute commands.
  • fs – to interact with the file system (reading and writing files).
  • path – to work with file and directory paths.
• Set Up Variables:
  • remoteFile – the path to the remote file we want to retrieve.
  • localFile – the name of the file to save locally (extracted using path.basename).
  • connection – a new instance of the SSH client.

Step 2: Establish SSH Connection and Execute SCP Command

connection.on('ready', () => {  
  connection.exec(`scp -f ${remoteFile}`, (err, stream) => {  
    if (err) {  
      console.error('Error executing SCP command:', err);  
      connection.end();  
      return;  
    }
    // The file transfer handling continues here
  });  
}).connect({
  host: 'your_remote_host',
  port: 22,
  username: 'your_username',
  privateKey: fs.readFileSync('/path/to/your/private/key')
});

• Establish Connection:
  • Use the connect method to establish an SSH connection to the server.
  • Provide the necessary connection parameters: host, port, username, and privateKey (or password).
• Handle the ready Event:
  • When the connection is established, the ready event is triggered.
  • Inside this event, execute the command scp -f ${remoteFile} using the exec method.
  • The -f option tells the server that the client wants to retrieve a file (from).

Step 3: Send the First Acknowledgment Byte 0x00

// Function to send acknowledgment  
const sendByte = (byte) => {  
  stream.write(Buffer.from([byte]));  
};  

// Send the initial acknowledgment byte  
sendByte(0);  

• Send Byte 0x00:
  • After sending the scp command, you need to send the byte 0x00 to synchronize with the server.
  • This signals to the server that the client is ready to receive data.

Step 4: Handle Incoming Data from the Server

let fileStream;  
let fileSize = 0;  
let receivedBytes = 0;  
let expect = 'response'; // Current expected state  
let buffer = Buffer.alloc(0);  

stream.on('data', (data) => {  
  buffer = Buffer.concat([buffer, data]);  
  while (true) {  
    if (buffer.length < 1) break;  
    // Handling different states
  }  
});  

• State Variables:
  • fileStream – the stream for writing the file.
  • fileSize – the size of the file obtained from the metadata.
  • receivedBytes – the number of bytes received from the server.
  • expect – the current expected state of the protocol.
  • buffer – buffer to store incoming data.
• Data Handling:
  • When data is received from the server, it is added to the buffer.
  • Use a while loop to process all available data in the buffer according to the current expected state.

Step 5: Handle SCP Protocol States

State 'response':

if (expect === 'response') {  
  const response = buffer[0];  
  if (response === 0x43) {  
    expect = 'metadata';  
  } else if (response === 0x01 || response === 0x02) {  
    expect = 'error';  
    buffer = buffer.slice(1);  
  } else {  
    console.error('Unknown server response:', response.toString(16));  
    connection.end();  
    return;  
  }  
}

• Handle Server Response:
  • Read the first byte from the buffer.
  • If the byte equals 0x43 (the ASCII character 'C'), it indicates the start of the file metadata; transition to the 'metadata' state.
  • If the byte equals 0x01 or 0x02, it signals an error; transition to the 'error' state.
  • Otherwise, log an error message and end the connection.

State 'metadata':

else if (expect === 'metadata') {  
  const nlIndex = buffer.indexOf(0x0A); // '\n'  
  if (nlIndex === -1) break; // Waiting for more data  

  const metadata = buffer.slice(0, nlIndex).toString();  
  buffer = buffer.slice(nlIndex + 1);  

  if (metadata.startsWith('C')) {  
    const parts = metadata.split(' ');  
    if (parts.length < 3) {  
      console.error('Error: Invalid metadata:', metadata);  
      connection.end();  
      return;  
    }  

    fileSize = parseInt(parts[1], 10);  
    if (isNaN(fileSize)) {  
      console.error('Error: Invalid file size in metadata:', metadata);  
      connection.end();  
      return;  
    }  

    // Create a write stream for the file  
    fileStream = fs.createWriteStream(localFile);  
    fileStream.on('error', (fileErr) => {  
      console.error('Error writing file:', fileErr);  
      connection.end();  
    });  

    // Send acknowledgment  
    sendByte(0);  

    expect = 'data';  
    receivedBytes = 0;  
    console.log('Starting file transfer...');  
  } else {  
    console.error('Error: Expected metadata line, received:', metadata);  
    connection.end();  
    return;  
  }  
}

• Handle File Metadata:
  • Search for the newline character \n in the buffer.
  • Extract the metadata string and remove it from the buffer.
  • Check that the string starts with 'C'.
  • Split the string into parts and extract the file size.
  • Validate the file size.
  • Create a write stream for the file.
  • Send the acknowledgment byte 0x00.
  • Transition to the 'data' state.

State 'data':

else if (expect === 'data') {  
  if (receivedBytes < fileSize) {  
    const remainingBytes = fileSize - receivedBytes;  
    const bytesToRead = Math.min(buffer.length, remainingBytes);  
    fileStream.write(buffer.slice(0, bytesToRead));  
    receivedBytes += bytesToRead;  
    buffer = buffer.slice(bytesToRead);  
    if (receivedBytes === fileSize) {  
      expect = 'data_response';  
    }  
  } else {  
    expect = 'data_response';  
  }  
}

• Receive File Data:
  • Calculate how many bytes remain to be received.
  • Read available data from the buffer and write it to the file.
  • Update the received bytes counter and the buffer.
  • If the entire file is received, transition to the 'data_response' state.

State 'data_response':

else if (expect === 'data_response') {  
  console.log('Bytes received:', receivedBytes);  
  if (buffer.length < 1) break; // Waiting for more data  

  const response = buffer[0];  
  buffer = buffer.slice(1);  

  if (response === 0) {  
    fileStream.end(() => {  
      console.log(`File ${localFile} saved successfully.`);  
    });  
    expect = 'end';  
    sendByte(0); // Send acknowledgment to finish  
  } else if (response === 1 || response === 2) {  
    expect = 'error';  
  } else {  
    console.error('Unknown server response after data transfer:', response);  
    connection.end();  
    return;  
  }  
}

• Handle Acknowledgment After Receiving Data:
  • Read the response byte from the server.
  • If the byte equals 0x00, finish writing the file and send acknowledgment.
  • If the byte equals 0x01 or 0x02, transition to the 'error' state.
  • Otherwise, log an error message and end the connection.

State 'end':

else if (expect === 'end') {  
  // Transfer completed  
  connection.end();  
  return;  
}

• Complete the Transfer:
  • Close the SSH connection.

State 'error':

else if (expect === 'error') {  
  const nlIndex = buffer.indexOf(0x0A); // '\n'  
  if (nlIndex === -1) break; // Waiting for more data  

  const errorMsg = buffer.slice(0, nlIndex).toString();  
  console.error(`SCP Error: ${errorMsg}`);  
  buffer = buffer.slice(nlIndex + 1);  
  connection.end();  
  return;  
}

• Handle Errors:
  • Read the error message from the server up to the \n character.
  • Log the error message.
  • End the connection.

Step 6: Handle Completion and Error Events

stream.on('close', () => {  
  console.log('Transfer completed.');  
  connection.end();  
});

stream.stderr.on('data', (data) => {  
  console.error(`STDERR: ${data.toString()}`);  
});

• close Event:
  • Log a message indicating the transfer is complete.
  • Close the connection.
• stderr Event:
  • Handle error messages received from the server.

Key Points to Consider

1. Synchronization with the Server. Sending the 0x00 byte at the correct moments (after sending the command, after receiving metadata, after receiving data) ensures proper synchronization and prevents hangs.
2. State Management. Explicitly tracking the current protocol state allows correct handling of different transfer stages and differentiates between file data and control signals.
3. Data Buffering. Use a buffer to accumulate incoming data to correctly handle cases where data arrives in chunks.
4. Error Handling. Properly handling potential errors and messages from the server is crucial for application reliability.

Conclusion

In this example, we implemented a basic file transfer mechanism using the SCP protocol in Node.js with the ssh2 package. We walked through each step of the process, from establishing the connection to completing the file transfer. Understanding how the SCP protocol works and correctly implementing message exchanges allows you to create a reliable solution for secure file transfer when using SFTP is not possible.

Important: This code is a basic example and may require additional enhancements and error handling for production use. It is recommended to add exception handling and logging to improve application reliability and debugging.

Links

• https://github.com/openssh/openssh-portable/blob/master/scp.c