Playing with raw image bytes

Hello, guys today let's play with image raw bytes. What I am gonna do is, manipulate the image raw pixel bytes, and make something silly, this is fun stuff to do. This article contains some theory as well as practical implementation, So let's go…

As we know an image is formed by a bunch of pixels together, pixel is nothing but a combination of RGB (Red, Green, Blue) or RGBA (Red, Green, Blue, Alpha) each with take 1 byte.

The images we view with extensions like PNG or JPG are the compressed formats of the image, PNG is lossless that PNG uses algorithms like DEFLATE to compress without losing the pixel and JPG is lossy compression that it will lose some pixel so there will be some loss in the image quality, if we want to view the image without the compression we need to convert the image to the BMP (Bitmap Image File) or there are also some other formats, if we convert to this we get the uncompressed image. But we don't need this we will extract those raw bytes and play with them and we will again convert them back to PNG or JPG.

First, let's set the client to upload images, I will set a simple react application for this

import axios from "axios";
import { useState } from "react";
import "./App.css";

function App() {
  const [img, setImg] = useState(null);
  const [pending, setPending] = useState(false);

  const handleImg = (e) => {
    setImg(e.target.files[0]);
  };

  const handleSubmit = async (e) => {
    e.preventDefault();
    if (!img) return;

    const formData = new FormData();
    formData.append("img", img);

    try {
      setPending(true);
      const response = await axios.post("http://localhost:4000/img", formData);
      console.log(response.data);

    } catch (error) {
      console.log("Error uploading image:", error);
    } finally {
      setPending(false);
    }
  };

  return (
    <div className="app-container">
      <div className="form-container">
        <form onSubmit={handleSubmit}>
          <input type="file" name="img" accept="image/*" onChange={handleImg} />
          <br />
          <button type="submit" disabled={pending}>
            {pending ? "Uploading..." : "Upload Image"}
          </button>
        </form>
      </div>
    </div>
  );
}

export default App;

So this is simple code, to upload the images, the main part is on the server side.

Now let's manually calculate the image bytes and check with the server-side code.

I have chosen the below image.

The image is from my previous article thumbnail. So this is a PNG file, if we go to the properties section, we can see the width and the height of the image. For this the width and the height are 722 x 407 which is equal to 293854 pixels, also this is not a total number of bytes, it is just a total number of pixels. As we know each pixel is either 3 or 4 bytes, RGB or RGBA. So if the above image is RGB the total bytes would be 722 x 407 x 3 = 881562 or if the image has the alpha channel, then the total bytes would be 722 x 407 x 4 = 1175416.

Let's some to the server side, I am using the node js.

There is a library called multer to parse multiform data.

app.post("/img", upload.single("img"), async (req, res) => {
  const arr = req.file.buffer
  console.log(arr.length)    //output: 30929
  res.send("success")
});

We store the image bytes in the buffer array, if we take the length of the buffer array the answer is 30929, there are these many bytes in the array, but wait the total number of bytes should be 1175416 right? What happens here is multer doesn't do some compression or anything, it just gets the image from the user and stores it in the buffer as it is, so we uploaded the PNG file, the buffer you are seeing is the same size as the PNG image size.

Now let's change the bytes in the compressed image byte.

app.post("/img", upload.single("img"), async (req, res) => {
  const arr = req.file.buffer;
  console.log("multer " + arr.length);
  fs.writeFile("output.png", arr, (err) => {
    console.log(err);
  });
  res.send("successfull");
});

I used the fs to create a new image with the existing one. So now if we change the first-byte arr[0] = 231, the image will not open.

Because the first certain bytes are reserved for the metadata, so if we change those metadata, and then the image can corrupt.

So let's jump to the 500th byte. arr[500] = 123, then write the image. But now, the image is broke, we should not directly manipulate the compressed image bytes because it can change the compression algorithm encoded data.

We need the raw bytes from the image, and then we can independently manipulate the bytes, and for that, we can use a sharp library.

npm install sharp

install the sharp, Now I will create a separate file to handle those logics,

sharp.js

export async function convert(buffer) {
  try {
    const data = await sharp(buffer).metadata();
    console.log(data)
  }catch(err){
    console.log(err)
  }
}

This is an async function, Now let's get the metadata from the png we have uploaded.

{
  format: 'png',
  size: 30929,
  width: 722,
  height: 407,
  space: 'srgb',
  channels: 4,
  depth: 'uchar',
  density: 72,
  isProgressive: false,
  hasProfile: false,
  hasAlpha: true
}

This is the metadata from the image, as we can see the last data hasAlpha: true so it has the alpha channel, so each pixel is 4 bytes.

Now let's get the raw bytes from the image.

const rawBytes = await sharp(buffer)
      .raw()
      .toBuffer({ resolveWithObject: true });

console.log(rawBytes.data.length)  //1175416

Now we can see the array length is equal to our calculation. So this image contains 1175416 bytes. Now we are free.. to change any bytes, Now the metadata is not stored in the buffer, the buffer only contains the raw bytes of the image.

Let's change only one pixel to red.

  rawBytes.data[0] = 225;    //red
  rawBytes.data[1] = 10;     //green
  rawBytes.data[2] = 10;     //blue
  rawBytes.data[3] = Math.floor(0.8 * 255);   //alpha

As we can one pixel is changed to red, we need to zoom in on the image to see the pixel change.

Now let's divide the image and change the color, the top half is yellow and the bottom half is green

const div = rawBytes.data.length / 2;
    for (let i = 0; i < rawBytes.data.length; i += 4) {
      if (i <= div) {
        rawBytes.data[i] = 240;
        rawBytes.data[i + 1] = 255;
        rawBytes.data[i + 2] = 21;
        rawBytes.data[i + 3] = Math.floor(0.8 * 255);
      } else {
        rawBytes.data[i] = 84;
        rawBytes.data[i + 1] = 135;
        rawBytes.data[i + 2] = 21;
        rawBytes.data[i + 3] = Math.floor(0.9 * 255);
      }
    }

We are incrementing the loop by 4 times because we are changing one pixel at each iteration. Now the output will be like this.

We can see the transparency in this image because the Alpha channel is set to 0.8

I forgot to tell for writing the image, we don't need fs to write a new image, we can use the sharp itself.

await sharp(rawBytes.data, {
      raw: {
        width: data.width,
        height: data.height,
        channels: data.channels,
      },
    })
      .png()
      .toFile("demo.png");

we are generating the new image with the same metadata.

Here's the full server side code,

//index.js
import express from "express";
import dotenv from "dotenv";
import multer from "multer";
import cors from "cors";
import { convert } from "./sharp.js";

const app = express();
dotenv.config();
app.use(cors({ origin: "http://localhost:5173" }));
const storage = multer.memoryStorage();
const upload = multer();

app.post("/img", upload.single("img"), async (req, res) => {
  const arr = req.file.buffer;
  await convert(arr);
  res.send("successful");
});

app.listen(process.env.PORT, () => {
  console.log("server started");
});

//sharp.js
import sharp from "sharp";

export async function convert(buffer) {
  try {
    const data = await sharp(buffer).metadata();
    console.log(data);
    //raw data
    const rawBytes = await sharp(buffer)
      .raw()
      .toBuffer({ resolveWithObject: true });
    console.log(rawBytes.data.length);
    const div = rawBytes.data.length / 2;
    for (let i = 0; i < rawBytes.data.length; i += 4) {
      if (i <= div) {
        rawBytes.data[i] = 240;
        rawBytes.data[i + 1] = 255;
        rawBytes.data[i + 2] = 21;
        rawBytes.data[i + 3] = Math.floor(0.8 * 255);
      } else {
        rawBytes.data[i] = 84;
        rawBytes.data[i + 1] = 135;
        rawBytes.data[i + 2] = 21;
        rawBytes.data[i + 3] = Math.floor(0.9 * 255);
      }
    }
    await sharp(rawBytes.data, {
      raw: {
        width: data.width,
        height: data.height,
        channels: data.channels,
      },
    })
      .png()
      .toFile("demo.png");
  } catch (error) {
    console.log(error.message);
  }
}

So this is it, We just played with those pixels. and finally this article thumbnail is made with this one line in the loop.

rawBytes.data[i] = Math.floor(Math.random()*256)

I just randomly changed each byte 😂

for the full code check out my repo: pixel-byte-manipulation

if there are any mistakes kindly do comment

Thank you!!!