Recently I was working on a project to backup files via the Autodesk BIM360/Construction Cloud API.
My initial prototype took 12 hours to backup roughly 100 GB (40,230 files in 11,737 folders) which wasn't good. Due to the fast pace data is being added to the account, within 1–2 years the backup would be taking over 24 hours.
After a program re-write which included moving the file downloading logic from a synchronous foreach loop to a Parallel.ForEachAsync loop I was able to get the nightly backup time down to 3 hours.
I wanted to quantify how much of the 4x performance improvement was down to the re-write vs the Parallel.ForEachAsync loop so ran two benchmarks in the awesome Benchmark.NET.
The first was on a small project containing 300 MB (136 files in 30 folders):
BenchmarkDotNet=v0.13.1, OS=Windows 10.0.19044.1645 (21H2)
Intel Core i7-5820K CPU 3.30GHz (Broadwell), 1 CPU, 12 logical and 6 physical cores
.NET SDK=6.0.202
[Host] : .NET 6.0.4 (6.0.422.16404), X64 RyuJIT
Job-KSORIT : .NET 6.0.4 (6.0.422.16404), X64 RyuJIT
IterationCount=3 LaunchCount=1 WarmupCount=1
| Method | Mean | Error | StdDev |
|---------------- |--------:|--------:|-------:|
| ParallelLoop | 123.2 s | 61.06 s | 3.35 s |
| SynchronousLoop | 299.8 s | 74.74 s | 4.10 s |
The second was on a large project containing 26 GB (10,653 files in 973 folders):
BenchmarkDotNet=v0.13.1, OS=Windows 10.0.19044.1706 (21H2)
Intel Core i7-5820K CPU 3.30GHz (Broadwell), 1 CPU, 12 logical and 6 physical cores
.NET SDK=6.0.202
[Host] : .NET 6.0.4 (6.0.422.16404), X64 RyuJIT
Job-QIVIZX : .NET 6.0.4 (6.0.422.16404), X64 RyuJIT
IterationCount=3 LaunchCount=1 RunStrategy=Monitoring
WarmupCount=0
| Method | Mean | Error | StdDev |
|---------------- |---------:|---------:|---------:|
| ParallelLoop | 98.83 m | 12.97 m | 0.711 m |
| SynchronousLoop | 300.62 m | 368.38 m | 20.192 m |
So the improvement from Parallel.ForEachAsync is roughly 3x.
I would have loved to run more iterations of benchmark 2 to get the error and stddev down but it took a day to run as is and the results match with both the benchmark 1 and the nightly real-world performance of the 100 GB backup.
Note the 100~ minutes for 26 GB backup in the benchmark above is using my home internet which is much slower than where the backup is running from each night.
The actual benchmark runner:
using ACC.ApiClient;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Engines;
using Library.Logger;
using Library.SecretsManager;
[SimpleJob(RunStrategy.Monitoring, launchCount: 1, warmupCount: 0, targetCount: 3)]
public class DownloadSynchronousVsParallel
{
private readonly ApiClient _apiClient;
private readonly string _projectId =
SecretsManager.GetEnvironmentVariable("acc:benchmarks:DownloadSynchronousVsParallel:projectId");
private readonly string _rootDirectory = Path.Combine(Path.GetTempPath(), Guid.NewGuid().ToString());
public DownloadSynchronousVsParallel()
{
string clientId = SecretsManager.GetEnvironmentVariable("acc:clientid");
string clientSecret = SecretsManager.GetEnvironmentVariable("acc:clientsecret");
string accountId = SecretsManager.GetEnvironmentVariable("acc:accountid");
_apiClient = TwoLeggedApiClient
.Configure()
.WithClientId(clientId)
.AndClientSecret(clientSecret)
.ForAccount(accountId)
.WithOptions(options =>
{
options.Logger = new NLogLogger(new NLogLoggerConfiguration
{
LogLevel = LogLevel.Debug,
LogToConsole = true
});
})
.Create();
}
[Benchmark]
public async Task ParallelLoop()
{
var project = await _apiClient.GetProject(_projectId);
await project.GetContentsRecursively();
await project.DownloadContentsRecursively(_rootDirectory);
Directory.Delete(_rootDirectory, true);
}
[Benchmark]
public async Task SynchronousLoop()
{
var project = await _apiClient.GetProject(_projectId);
await project.GetContentsRecursively();
foreach (var file in project.FilesRecursive)
await _apiClient.DownloadFile(file, _rootDirectory);
Directory.Delete(_rootDirectory, true);
}
}
The SynchronousLoop task on line 49 does what you expect, a foreach loop iterates over an IEnumerable of files and passes each down to a DownloadFile method:
public async Task<FileInfo> DownloadFile(
File file, string rootDirectory, CancellationToken ct = default)
{
CreateDirectory(file.ParentFolder, rootDirectory);
string downloadPath = Path.Combine(rootDirectory, file.GetPath()[1..]);
if (Config.DryRun)
{
await System.IO.File.WriteAllBytesAsync(downloadPath, Array.Empty<byte>(), ct);
file.FileInfo = new FileInfo(downloadPath);
file.FileSizeOnDisk = file.FileInfo.Length;
Config.Logger?.Info($"{file.FileInfo.FullName} ({file.FileSizeOnDiskInMb} MB)");
return file.FileInfo;
}
return await Config.RetryPolicy.ExecuteAsync(async () =>
{
await EnsureAccessToken();
file.DownloadAttempts++;
await using Stream stream = await Config.HttpClient.GetStreamAsync(file.DownloadUrl, ct);
await using FileStream fileStream = new(downloadPath, FileMode.Create);
await stream.CopyToAsync(fileStream, ct);
file.FileSizeOnDisk = fileStream.Length;
file.FileInfo = new FileInfo(downloadPath);
if (file.FileSizeOnDisk == file.StorageSize)
Config.Logger?.Debug($"{file.FileInfo.FullName} ({file.FileSizeOnDiskInMb} MB)");
else
Config.Logger?.Warn(
@$"{file.FileInfo.FullName} ({file.FileSizeOnDiskInMb}/{file.ApiReportedStorageSizeInMb} bytes) (Mismatch between size reported by API and size downloaded to disk)");
return file.FileInfo;
});
}
The file downloading logic is wrapped in a Polly AsyncRetryPolicy and is essentially 3 lines (20–22).
Both the SynchronousLoop and the ParallelLoop use the above method to download files.
The ParallelLoop’s ‘DownloadContentsRecursively’ call on line 44 of the benchmark uses the below method incorporating Parallel.ForEachAsync:
public async Task<List<FileInfo>> DownloadFiles(
IEnumerable<File> fileList, string rootDirectory, CancellationToken ct = default)
{
List<FileInfo> fileInfoList = new();
var parallelOptions = new ParallelOptions
{
CancellationToken = ct,
MaxDegreeOfParallelism = Config.MaxDegreeOfParallelism
};
await Parallel.ForEachAsync(fileList, parallelOptions, async (file, ctx) =>
{
FileInfo fileInfo = await DownloadFile(file, rootDirectory, ctx);
fileInfoList.Add(fileInfo);
});
return fileInfoList;
}
As you can see on line 14 the same DownloadFile method is being called as with the synchronous loop.
The C# dev team have really made it incredibly simple to incorporate parallelism into our applications when the situation calls for it.
Top comments (0)