An efficient way to send larger messages over the network

In this post, we will see how we can send large messages over the socket to reduce the bandwidth and increase the message throughput. We are always writing server applications to interact with clients back and worth, but most of the time, we just send the messages using open API application protocol like JSON.

If you are not worried about bandwidth and performance, we can directly send the message as it is, but that is not the case for every application and environment. Think about the following use-case.

Producer A generating JSON message in the following schema, where data is a big chunk of the binary message. Total message size is 665417 bytes.

{
  "op": "publish",
  "topic": "/map",
  "msg": {
    "info": {
      "origin": {
        "position": {
          "w": -21.2,
          "x": -10,
          "y": 0
        },
        "orientation": {
          "w": 0,
          "x": 0,
          "y": 0,
          "z": 1
        }
      },
      "width": 608,
      "height": 384,
      "resolution": 0.05,
      "map_load_time": {
        "secs": 0,
        "nsecs": 1000000
      }
    },
    "header": {
      "header_stamp": {
        "secs": 0,
        "nsecs": 1000000
      },
      "frame_id": "map",
      "seq": 0
    },
    "data": [
      -1,
      -1,
      0,
      0
    ]
  }
}

How can we send this message over the network to the client efficiently? The most successful method is to use the PNG compression, using that, we can selectively agree on the fields to compress or compress the entire message and re-packaged in a simplified JSON format.

PNG compression basically compresses any message into PNG format by finding the minimum width and height of the input data. We are going to use the 3 channels (BGR) to fill our data. For example, 3 channel arrays are like the following.

So, let's write our own PNGDataCompression function in C++.

      std::string ConvertToPNGEncoding(std::string _zDataMsg)
     {
        int iBufferLen = _zDataMsg.length();
        int iWidth = floor(sqrt(iBufferLen / 3.0));
        int iHeight = ceil((iBufferLen / 3.0) / iWidth);
        // we need to calculate how much space is left to fill to complete
        // entire data structure.
        int iBytesNeeded = int(iWidth * iHeight * 3);

        // let's fill remaining space with new lines.
        while (iBufferLen < iBytesNeeded)
        {
                _zDataMsg.append("\n");
                iBufferLen += 1;
        }

        // now lets encode the entire buffer using opencv png encoding.
        std::vector<uint8_t> vecRAWBuffer;
        vecRAWBuffer.assign(_zDataMsg.begin(), _zDataMsg.end());

        cv::Mat imgOriginalData(iWidth, iHeight, CV_8UC3);
        imgOriginalData.data = vecRAWBuffer.data();

        cv::Mat imgGreyImage;
        cv::cvtColor(imgOriginalData, imgGreyImage, CV_BGR2RGB);

        std::vector<uint8_t> vecOfEncodedBuffer;
        std::vector<int> encode_params;
        encode_params.push_back(CV_IMWRITE_PNG_COMPRESSION);
        cv::imencode(".png", imgGreyImage, vecOfEncodedBuffer, encode_params);

         // now we have the binary data, in order to send through the standard JSON, we will apply base64 encoding. 


          string zPNGEncoded(vecOfEncodedBuffer.begin(), vecOfEncodedBuffer.end());

        std::string encodedData = StandardBase64Encode(zPNGEncoded.c_str(),zPNGEncoded.length());

         return encodedData;
     }


// standared base64 encoding. 
     std::string StandardBase64Encode(const char * _zEncodingBuffer, unsigned int _iBufferLen) {
  std::string ret;
  int i = 0;
  int j = 0;
  unsigned char char_array_3[3];
  unsigned char char_array_4[4];

  while (_iBufferLen--) {
    char_array_3[i++] = *(_zEncodingBuffer++);
    if (i == 3) {
      char_array_4[0] = (char_array_3[0] & 0xfc) >> 2;
      char_array_4[1] = ((char_array_3[0] & 0x03) << 4) + ((char_array_3[1] & 0xf0) >> 4);
      char_array_4[2] = ((char_array_3[1] & 0x0f) << 2) + ((char_array_3[2] & 0xc0) >> 6);
      char_array_4[3] = char_array_3[2] & 0x3f;

      for(i = 0; (i <4) ; i++)
        ret += zBase64Chars[char_array_4[i]];
      i = 0;
    }
  }

  if (i)
  {
    for(j = i; j < 3; j++)
      char_array_3[j] = '\0';

    char_array_4[0] = (char_array_3[0] & 0xfc) >> 2;
    char_array_4[1] = ((char_array_3[0] & 0x03) << 4) + ((char_array_3[1] & 0xf0) >> 4);
    char_array_4[2] = ((char_array_3[1] & 0x0f) << 2) + ((char_array_3[2] & 0xc0) >> 6);
    char_array_4[3] = char_array_3[2] & 0x3f;

    for (j = 0; (j < i + 1); j++)
      ret += zBase64Chars[char_array_4[j]];

    while((i++ < 3))
      ret += '=';
  }

  return ret;
}

As a result, the original message size is 665417, it will be reduced to just 10500. So, we can construct our new message like following

   {
     "op":"png"  /*what is the compression we used, so that client can use is accordingly */
     "data":"iVBORw0KGgoAAAANSUhEUgAA...."
   }

So, servers heavily sending large messages to the client can use the above technique to publish messages.

Python has all the necessary libraries to implement the above technique by using a couple of imports.

from PIL import Image
from base64 import standard_b64encode, standard_b64decode
from math import floor, ceil, sqrt


def pngencode(string):

    length = len(string)
    width = floor(sqrt(length/3.0))
    height = ceil((length/3.0) / width)
    bytes_needed = int(width * height * 3)
    while length < bytes_needed:
        string += '\n'
        length += 1
    i = Image.frombytes('RGB', (int(width), int(height)), string)
    buff = StringIO()
    i.save(buff, "png")
    encoded = standard_b64encode(buff.getvalue())
    return encoded

I hope this will help serverside programmers to implement png compression over larger messages.