In this article, we review the File upload mechanism in Immich codebase. We will look at:
fileUploadHandler.
uploadExecutionQueue.
fileUploader.
In part 1, we learnt what is Immich, locating the upload button in the codebase and reviewed the utils function, openFileUploadDialog. In this part 2, we get to the actual uploading process.
fileUploadHandler
fileUploadHandler is defined as shown below
export const fileUploadHandler = async ({
files,
albumId,
isLockedAssets = false,
}: FileUploadHandlerParams): Promise<string[]> => {
const extensions = uploadManager.getExtensions();
const promises = [];
for (const file of files) {
const name = file.name.toLowerCase();
if (extensions.some((extension) => name.endsWith(extension))) {
const deviceAssetId = getDeviceAssetId(file);
uploadAssetsStore.addItem({ id: deviceAssetId, file, albumId });
promises.push(
uploadExecutionQueue.addTask(
() => fileUploader({ assetFile: file, deviceAssetId, albumId, isLockedAssets })
),
);
}
}
const results = await Promise.all(promises);
return results.filter((result): result is string => !!result);
};
We know extenstions is an array of supported types from part 1. getDeviceAssetId simply returns a dynamic string.
function getDeviceAssetId(asset: File) {
return 'web' + '-' + asset.name + '-' + asset.lastModified;
}
uploadAssetsStore is imported as shown below:
import { uploadAssetsStore } from '$lib/stores/upload';
In the stores/upload, you will see this:
import { UploadState, type UploadAsset } from '$lib/types';
import { derived, writable } from 'svelte/store';
function createUploadStore() {
const uploadAssets = writable<Array<UploadAsset>>([]);
const stats = writable<{ errors: number; duplicates: number; success: number; total: number }>({
errors: 0,
duplicates: 0,
success: 0,
total: 0,
});
...
const addItem = (newAsset: UploadAsset) => {
uploadAssets.update(($assets) => {
const duplicate = $assets.find((asset) => asset.id === newAsset.id);
if (duplicate) {
return $assets.map((asset) => (asset.id === newAsset.id ? newAsset : asset));
}
stats.update((stats) => {
stats.total++;
return stats;
});
$assets.push({
...newAsset,
speed: 0,
state: UploadState.PENDING,
progress: 0,
eta: 0,
});
return $assets;
});
};
...
}
export const uploadAssetsStore = createUploadStore();
uploadExecutionQueue
At L44, uploadExecutionQueue is initialized:
export const uploadExecutionQueue = new ExecutorQueue({ concurrency: 2 });
ExecutorQueue
ExecutorQueue is an algorithm that uses an array to push queue items and uses concurrency.
import { handlePromiseError } from '$lib/utils';
interface Options {
concurrency: number;
}
type Runnable = () => Promise<unknown>;
export class ExecutorQueue {
private queue: Array<Runnable> = [];
private running = 0;
private _concurrency: number;
constructor(options?: Options) {
this._concurrency = options?.concurrency || 2;
}
get concurrency() {
return this._concurrency;
}
set concurrency(concurrency: number) {
if (concurrency < 1) {
return;
}
this._concurrency = concurrency;
const v = concurrency - this.running;
if (v > 0) {
for (let i = 0; i < v; i++) {
this.tryRun();
}
}
}
addTask<T>(task: () => Promise<T>): Promise<T> {
return new Promise((resolve, reject) => {
// Add a custom task that wrap the original one;
this.queue.push(async () => {
try {
this.running++;
const result = task();
resolve(await result);
} catch (error) {
reject(error);
} finally {
this.taskFinished();
}
});
// Then run it if possible !
this.tryRun();
});
}
private taskFinished(): void {
this.running--;
this.tryRun();
}
private tryRun() {
if (this.running >= this.concurrency) {
return;
}
const runnable = this.queue.shift();
if (!runnable) {
return;
}
handlePromiseError(runnable());
}
}
You see this.queue.shift? that pops the first items in the queue and runs it using handlePromiseError(runnable()). This is neat.
fileUploader
uploadExecutionQueue.addTask(() => fileUploader({ assetFile: file, deviceAssetId, albumId, isLockedAssets })),
fileUploader function is the actual task that is run to upload files. This function is large, so I will review what is going on at high level.
checkBulkUpload
Before the actual upload, there is a check in place. One example I know is that in the demo URL of Immich, if you try to upload, it is rejected since it is demo.
try {
const bytes = await assetFile.arrayBuffer();
const hash = await crypto.subtle.digest('SHA-1', bytes);
const checksum = Array.from(new Uint8Array(hash))
.map((b) => b.toString(16).padStart(2, '0'))
.join('');
const {
results: [checkUploadResult],
} = await checkBulkUpload({ assetBulkUploadCheckDto: { assets: [{ id: assetFile.name, checksum }] } });
if (checkUploadResult.action === Action.Reject && checkUploadResult.assetId) {
responseData = {
status: AssetMediaStatus.Duplicate,
id: checkUploadResult.assetId,
isTrashed: checkUploadResult.isTrashed,
};
}
} catch (error) {
console.error(`Error calculating sha1 file=${assetFile.name})`, error);
}
uploadRequest
If there is no issues with checks, then the uploadRequest is invoked
if (!responseData) {
const queryParams = asQueryString(authManager.params);
uploadAssetsStore.updateItem(deviceAssetId, { message: $t('asset_uploading') });
const response = await uploadRequest<AssetMediaResponseDto>({
url: getBaseUrl() + '/assets' + (queryParams ? `?${queryParams}` : ''),
data: formData,
onUploadProgress: (event) => uploadAssetsStore.updateProgress(deviceAssetId, event.loaded, event.total),
});
if (![200, 201].includes(response.status)) {
throw new Error($t('errors.unable_to_upload_file'));
}
responseData = response.data;
}
uploadRequest is defined in lib/utils.ts as shown below:
export const uploadRequest = async <T>(
options: UploadRequestOptions
): Promise<{ data: T; status: number }> => {
const { onUploadProgress: onProgress, data, url } = options;
return new Promise((resolve, reject) => {
const xhr = new XMLHttpRequest();
xhr.addEventListener('error', (error) => reject(error));
xhr.addEventListener('load', () => {
if (xhr.readyState === 4 && xhr.status >= 200 && xhr.status < 300) {
resolve({ data: xhr.response as T, status: xhr.status });
} else {
reject(new ApiError(xhr.statusText, xhr.status, xhr.response));
}
});
if (onProgress) {
xhr.upload.addEventListener('progress', (event) => onProgress(event));
}
xhr.open(options.method || 'POST', url);
xhr.responseType = 'json';
xhr.send(data);
});
};
So Immich uses XMLHttpRequest, no axios or fetch.
About me:
Hey, my name is Ramu Narasinga. I study codebase architecture in large open-source projects.
Email: ramu.narasinga@gmail.com
I spent 200+ hours analyzing Supabase, shadcn/ui, LobeChat. Found the patterns that separate AI slop from production code. Stop refactoring AI slop. Start with proven patterns. Check out production-grade projects at thinkthroo.com
Top comments (0)