DEV Community

Cover image for HEIC images and the web
Tarwin Stroh-Spijer
Tarwin Stroh-Spijer

Posted on

HEIC images and the web

#javascript #ios #webdev #programming

HEIC is an image format, really only used by iOS.

Most browsers do not support the format and this is unlikely to change.

So why do I care about them? Because sometimes people just require that they can upload HEIC files they have sent from their iPhones, or an app on an iPhone works with HEIC.

Gotchas!

Image rotations

The biggest frustration I've found is that sometimes data about image orientation is stored both in the EXIF Orientation tag, as well as in the HEIC ItemPropertyContainerBox itself as an irot.

What does this mean? Well the EXIF Orientation could be set at 6 - which means "Rotated 90° counterclockwise (270° clockwise)". And then the irot rotation could be set to 1 - which means "rotate 90° clockwise'. So you end up with no rotation!

From what I've found, the irot rotation should only be applied if an EXIF Orientation is set at all. Otherwise it can be ignored. This could simply be because of the HEIC files I have for testing.

Windows

Of course there are extra problems when using Windows. It seems that sometimes Windows decides not to inform the browser of the correct mime type when adding HEIC files. Why? Who knows! But it does mean you will have to check for .heic extensions as well as mime types.

Loading in the browser

Introduction

There is a great library, heic2any, which I use to convert the HEIC image data to a JPEG, for preview.

And then I use ExifReader, to read EXIf data.

There is also some code I wrote myself (with Claude's help) which you will find at the end of the article, HeicRotationExtractor, which extracts the irot data directly from the HEIC container, as the ExifReader doesn't do this.

The Code 

// imports
import heic2any from 'heic2any'
import ExifReader from 'exifreader'
import { getRotationFromHEIC } from './HeicRotationExtractor.ts'

const supportedMimeTypes = [
  { label: 'jpeg', mime: 'image/jpeg' },
  { label: 'png', mime: 'image/png' },
  { label: 'gif', mime: 'image/gif' },
  { label: 'webp', mime: 'image/webp' },
  { label: 'heic', mime: 'image/heic' },
]

const heicToJpeg = async (file:File) => {
  const blob = await heic2any({
    blob: file,
    toType: 'image/jpeg',
    quality: 0.5,
  })

  return URL.createObjectURL(Array.isArray(blob) ? blob[0] : blob)
}

// use this somewhere to display the media previews
const mediaToDisplay = []

const onFileChange = async (e:any) => {
  let errored
  let erroredMimeType = ''
  for (let i=0; i<e.target.files.length; i++) {
    const file = e.target.files[i] as File

    let isHeicOverride = false
    // HACK: windows sucks at HEIC files
    // if file ends in .heic, then it's a HEIC file 
    if (file.name.toLowerCase().endsWith('.heic')) {
      isHeicOverride = true
    }

    if (supportedMimeTypes.map(m => m.mime).includes(file.type) || isHeicOverride) {
      let url
      if (file.type === 'image/heic' || isHeicOverride) {
        let rotationAngle = 0
        let shouldFlip = false
        let shouldFlop = false

        try {
          url = await heicToJpeg(file)
        } catch (err) {
          console.error(err)
          alert('Failed to load HEIC image')
          continue
        }

        try {
          const tags = await ExifReader.load(file)
          const orientation = tags?.Orientation?.value || 1

          // const containerRotation = await getHEICRotation(file)
          const containerRotation = await getRotationFromHEIC(file)

          // Calculate base rotation from EXIF orientation
          let baseRotation = 0;
          switch (orientation) {
            case 6: baseRotation = 90; break;
            case 3: baseRotation = 180; break;
            case 8: baseRotation = 270; break;
            case 2: shouldFlop = true; break;
            case 7: shouldFlip = true; baseRotation = 90; break;
            case 4: shouldFlop = true; baseRotation = 180; break;
            case 5: shouldFlip = true; baseRotation = 270; break;
          }

          // HACK: seems for some reason that it's only ones that have orientations that are set
          // that should even be considered for rotation
          // Combine both rotations
          if (tags?.Orientation?.value) {
            rotationAngle = (baseRotation + (containerRotation ?? 0)) % 360;
          }

          // Handle negative rotations
          if (rotationAngle < 0) {
            rotationAngle += 360;
          }

          // console.log('Orientation:', orientation, 'Container Rotation:', containerRotation, 'Rotation Angle:', rotationAngle)
        } catch (err) {
          console.error(err)
        }

        mediaToDisplay.push({
          url,
          file,
          displayMeta: {
            rotate: rotationAngle,
            flip: shouldFlip,
            flop: shouldFlop,
          }
        })
      } else {
        url = URL.createObjectURL(file)
        mediaToDisplay.push({
          url,
          file,
        })
      }
    } else {
      errored = true
      erroredMimeType = file.type
    }
  }
  if (errored) {
    alert('Unsupported Format', `Only ${supportedMimeTypes.map(m => m.label).join(', ')} images are supported. This file had a mime type of "${erroredMimeType}".`) 
  }
}
Enter fullscreen mode Exit fullscreen mode

To display you will need to do something like:

<img :src="media.url" :style="`transform: ${media.meta.rotate || 0}`">
Enter fullscreen mode Exit fullscreen mode

Transforming in Node

Introduction

Sharp is a great modern library for image manipulation in Node. It uses libvips under the hood and is super fast and awesome.

But of course it doesn't support HEIC by default. Why? Because it's a "patent-encumbered" format.

So if you want to run this you will need to install a version of libvips that does support HEIC.

My solution also requires ExifTool.

If you want to get this to work on Lambda you will need to add layers:

  • ExifTool
  • Python (for ExifTool)
  • Sharp (with included HEIC compatible libvips)

The Code

Get the Orienatation from the EXIF.

// some setup
let orientation = 1
let exifHasOrientationSet = false
let containerRotationAngle = null

// try and get orientation
try {
  let exifToolOrientation
  const output = await spawnAsync(`/opt/bin/perl`, [`/opt/bin/exiftool`, `-n`, `-Orientation`, `/tmp/image`])
  exifToolOrientation = parseInt(output.replace(/[^0-9]/g, ''))
  if (!isNaN(exifToolOrientation)) {
    exifHasOrientationSet = true
    orientation = exifToolOrientation
    console.log('"Orientation" from exiftool', orientation)
  }
} catch (e) {
  console.log(`ERROR: could not ready exif using 'exiftool'`, e)
}
Enter fullscreen mode Exit fullscreen mode

Because I use exiftool called from Node to get Orientation we can also use it to get Rotation, which it seems to do correctly from the container irot.

// for heic images, we may also need to get the "Rotation"
// it doesn't come from EXIF but from MPEG container
// but exiftool should get it correctly if it exists
// only need to do it if we have EXIF Orientation because
// if not we can ignore it
if (extension === 'heic' && exifHasOrientationSet) {
  try {
    const output = await spawnAsync(`/opt/bin/perl`, [`/opt/bin/exiftool`, `-n`, `-Rotation`, `/tmp/image`])
    const rotation = parseInt(output.replace(/[^0-9]/g, ''))
    if (!isNaN(rotation)) {
      containerRotationAngle = rotation
      console.log('"Rotation" from exiftool', containerRotationAngle)
    }
  } catch (e) {
    console.log(`ERROR: could not ready rotation using 'exiftool'`, e)
  }
}
Enter fullscreen mode Exit fullscreen mode

Do the rotation and orientation stuff.

let rotationAngle = 0
let shouldFlip = false
let shouldFlop = false
switch (orientation) {
  case 6: rotationAngle = 90; break;
  case 3: rotationAngle = 180; break;
  case 8: rotationAngle = 270; break;
  case 2: shouldFlop = true; break;
  case 7: shouldFlip = true; rotationAngle = 90; break;
  case 4: shouldFlop = true; rotationAngle = 180; break;
  case 5: shouldFlip = true; rotationAngle = 270; break;
}

if (exifHasOrientationSet && containerRotationAngle) {
  rotationAngle += containerRotationAngle
  // make sure it's between 0 and 360 and positive
  rotationAngle = rotationAngle % 360
  if (rotationAngle < 0) {
    rotationAngle = 360 + rotationAngle
  }
  console.log('rotationAngle updated with container rotation', rotationAngle)
}
Enter fullscreen mode Exit fullscreen mode

And apply to an instance of sharp

const resizedImageBuffer = await sharpInst
  .withMetadata()
  .resize({
    width: maxWidthHeightWeb,
    height: maxWidthHeightWeb,
    fit: 'inside',
  })
  .flip(shouldFlip)
  .flop(shouldFlop)
  .rotate(rotationAngle)
  .jpeg({ quality: 80 })
  .toBuffer();
Enter fullscreen mode Exit fullscreen mode

HeicRotationExtractor.ts 

const debug = false;
const debugPrint = (...args: any[]) => {
  if (debug) console.log(...args);
}

class HEICRotationExtractor {
  private view: DataView;

  constructor(buffer: ArrayBuffer) {
    this.view = new DataView(buffer);
  }

  async extractRotation(): Promise<number | null> {
    try {
      debugPrint('Looking for HEIC rotation...');

      // Find meta box
      const metaBox = this.findBox('meta');
      if (!metaBox) return null;

      debugPrint(`Found meta box at offset: ${metaBox.start.toString(16)}`);

      // Find ItemProperties (iprp) box within meta
      const iprpBox = this.findBoxInRange('iprp', metaBox.start + 12, metaBox.start + metaBox.size);
      if (!iprpBox) return null;

      debugPrint(`Found iprp box at offset: ${iprpBox.start.toString(16)}`);

      // Find ItemPropertyContainer (ipco) within iprp
      const ipcoBox = this.findBoxInRange('ipco', iprpBox.start + 8, iprpBox.start + iprpBox.size);
      if (!ipcoBox) return null;

      debugPrint(`Found ipco box at offset: ${ipcoBox.start.toString(16)}, size: ${ipcoBox.size}`);

      // Parse the properties in ipco box
      return this.parseItemPropertyContainer(ipcoBox.start, ipcoBox.size);

    } catch (error) {
      console.error('Error extracting rotation:', error);
      return null;
    }
  }

  private parseItemPropertyContainer(ipcoStart: number, ipcoSize: number): number | null {
    let offset = ipcoStart + 8; // Skip ipco box header
    const ipcoEnd = ipcoStart + ipcoSize;
    let propertyIndex = 1;

    debugPrint('Parsing ItemPropertyContainer properties...');

    while (offset < ipcoEnd - 8) {
      try {
        const propSize = this.view.getUint32(offset);
        const propType = this.getString(offset + 4, 4);

        debugPrint(`Property ${propertyIndex}: type='${propType}', size=${propSize}, offset=0x${offset.toString(16)}`);

        if (propSize === 0 || propSize > ipcoEnd - offset) {
          debugPrint('Invalid property size, breaking');
          break;
        }

        // Look for rotation in property #4 specifically
        if (propertyIndex === 4) {
          debugPrint(`Examining property #4 (${propType})...`);

          if (propType === 'irot') {
            // Image rotation property - single byte after box header
            const rotationValue = this.view.getUint8(offset + 8);
            debugPrint(`Found irot property with value: ${rotationValue}`);
            return rotationValue * 90;
          } else {
            // If it's not 'irot', dump the content to see what's there
            debugPrint(`Property #4 is '${propType}', examining content...`);
            this.dumpBoxContent(offset, Math.min(propSize, 32));

            // Look for a byte value of 3 in the property
            for (let i = 8; i < Math.min(propSize, 20); i++) {
              const value = this.view.getUint8(offset + i);
              if (value === 3) {
                debugPrint(`Found value 3 at byte offset ${i} in property #4`);
                return 270; // 3 * 90 degrees
              }
            }
          }
        }

        // Also check for any rotation properties regardless of position
        if (propType === 'irot') {
          const rotationValue = this.view.getUint8(offset + 8);
          debugPrint(`Found irot property at position ${propertyIndex} with value: ${rotationValue}`);
          return rotationValue * 90;
        }

        offset += propSize;
        propertyIndex++;
      } catch (e) {
        debugPrint(`Error parsing property ${propertyIndex}:`, e);
        break;
      }
    }

    return null;
  }

  private findBox(boxType: string): { start: number; size: number } | null {
    let offset = 0;

    while (offset <= this.view.byteLength - 8) {
      try {
        const size = this.view.getUint32(offset);
        const type = this.getString(offset + 4, 4);

        if (type === boxType) {
          return { start: offset, size };
        }

        if (size === 0 || size > this.view.byteLength - offset) {
          break;
        }

        offset += size;
      } catch (e) {
        break;
      }
    }

    return null;
  }

  private findBoxInRange(boxType: string, rangeStart: number, rangeEnd: number): { start: number; size: number } | null {
    let offset = rangeStart;

    while (offset <= rangeEnd - 8) {
      try {
        const size = this.view.getUint32(offset);
        const type = this.getString(offset + 4, 4);

        if (type === boxType) {
          return { start: offset, size };
        }

        if (size === 0 || size > rangeEnd - offset) {
          break;
        }

        offset += size;
      } catch (e) {
        break;
      }
    }

    return null;
  }

  private getString(offset: number, length: number): string {
    let result = '';
    for (let i = 0; i < length; i++) {
      if (offset + i >= this.view.byteLength) break;
      result += String.fromCharCode(this.view.getUint8(offset + i));
    }
    return result;
  }

  private dumpBoxContent(offset: number, length: number): void {
    debugPrint('Box content (hex):');
    let line = '';
    for (let i = 0; i < length; i++) {
      if (i % 16 === 0 && i > 0) {
        debugPrint(`  ${(offset + i - 16).toString(16).padStart(4, '0')}: ${line}`);
        line = '';
      }
      const byte = this.view.getUint8(offset + i);
      line += byte.toString(16).padStart(2, '0') + ' ';
    }
    if (line) {
      debugPrint(`  ${(offset + length - (length % 16)).toString(16).padStart(4, '0')}: ${line}`);
    }
  }

  // Debug method to show all properties in ipco
  debugAllProperties(): void {
    const metaBox = this.findBox('meta');
    if (!metaBox) return;

    const iprpBox = this.findBoxInRange('iprp', metaBox.start + 12, metaBox.start + metaBox.size);
    if (!iprpBox) return;

    const ipcoBox = this.findBoxInRange('ipco', iprpBox.start + 8, iprpBox.start + iprpBox.size);
    if (!ipcoBox) return;

    debugPrint('=== All ItemPropertyContainer Properties ===');
    let offset = ipcoBox.start + 8;
    const ipcoEnd = ipcoBox.start + ipcoBox.size;
    let propertyIndex = 1;

    while (offset < ipcoEnd - 8) {
      try {
        const propSize = this.view.getUint32(offset);
        const propType = this.getString(offset + 4, 4);

        debugPrint(`Property ${propertyIndex}: '${propType}' (${propSize} bytes) at 0x${offset.toString(16)}`);

        if (propSize === 0 || propSize > ipcoEnd - offset) break;

        offset += propSize;
        propertyIndex++;
      } catch (e) {
        break;
      }
    }
  }
}

// Usage
export async function getRotationFromHEIC(file: File): Promise<number | null> {
  const buffer = await file.arrayBuffer();
  const extractor = new HEICRotationExtractor(buffer);

  // First show all properties
  debugPrint('=== Debug: All Properties ===');
  extractor.debugAllProperties();

  debugPrint('=== Extracting Rotation ===');
  return await extractor.extractRotation();
}
Enter fullscreen mode Exit fullscreen mode

Top comments (0)