Transactional BLOB storage uploads with C#

The Background

The unit of work pattern is one that I nearly always use when writing a repository layer. It ensures that repository operations can be batched into arbitrary atomic units, meaning they all succeed or fail as one (i.e., it is impossible to finish in a ‘half completed’ state).

Implementing a unit of work involves setting up a transaction. This single transaction is a scope that wraps all repository operations that happen during its lifetime, and can be completed at will by the calling code (the unit of work is then responsible for automatically setting up a new transaction before any further repository operations take place). This saves having to start up transactions manually within our actual repositories and ensures that anything happening in those repositories is transactional by default.

Service-layer code then interacts with the unit of work, rather than using repositories directly (which would circumvent the ambient transaction).

SQL is fully geared up for supporting transactions, so it is relatively straightforward to make sure connections to a SQL database get enlisted in a transaction. However, this behaviour is often not natively supported by other forms of repository storage, such as file (BLOB) storage.

I recently had a bit of a go at writing some C# code to ensure that any BLOB storage uploads to Azure, happening within the context of a unit of work, are undone if the transaction doesn’t complete successfully.

This means we can have a unit of work that wraps both SQL calls and BLOB storage calls, resulting in the following possible chain of events:

Successful SQL UPDATE ✅
Successful BLOB storage file upload ✅
Failed SQL INSERT ❌
SQL UPDATE rolled back ✅
BLOB storage file deleted ✅

I find this very pleasing, since even though BLOB storage is cheap it’s nice not to have files clogging it up that shouldn’t be there!

The Code

I decided to use the Command Pattern to model the BLOB upload and deletion behaviour I was looking for.

This pattern models a single operation as a standalone object, with one of the benefits being that you can specify the code that defines an operation’s execution and also the code that specifies how to undo it.

Here’s the basic interface I used:

public interface ICommand<T> 
{
    Task<T> Execute();
    Task<T> Undo();
}

With this, I can create an UploadFileCommand class to model the operation of executing and undoing a file upload:

public class UploadFileCommand<T> : ICommand<T> where T : IBlobFile
{
    private readonly BlobStorageRepository<T> _repository;
    private readonly T _file;

    public UploadFileCommand(BlobStorageRepository<T> repository, T file)
    {
        _repository = repository;
        _file = file;
    }

    // Execute = upload my file
    public Task<T> Execute() => _repository.UploadFile(_file);
    
    // Undo = delete that same file
    public Task<T> Undo() => _repository.DeleteFile(_file);   
}

As stated in the code comments above, Execute() uploads the BLOB file and Undo() deletes that same file. This file has already been passed in via the constructor so an instantiated UploadFileCommand is always operating on the exact same file.

The other constructor argument is the repository that the command is going to use to upload and delete the file. In my example it’s typed to T so we can enforce that it deals with the exact type of file that we expect it to.

The unit of work is set up as follows (I’m only showing the relevant bits here - leaving out the boilerplate disposal etc. for brevity):

public sealed class UnitOfWork : IUnitOfWork, IDisposable
{
    private SqlConnection? _connection;
    private readonly ITransactionScopeFactory _transactionScopeFactory;

    private TransactionScope? _transactionScope;
    private TransactionScope? TransactionScope
    {
        get => _transactionScope;
        set
        {
            if (_transactionScope == value) return;
            _transactionScope = value;

            // Rollback any uncommitted BLOB uploads when starting new transaction or disposing existing one
            RollbackBlobUploads();

            if (_transactionScope == null) return;
            _connection?.EnlistTransaction(Transaction.Current);
        }
    }

    ... // other SQL repos

    private readonly Lazy<IBlobStorageRepository<PdfFile>> _pdfRepository;
    private readonly Lazy<IBlobStorageRepository<CsvFile>> _csvRepository;

    public IBlobStorageRepository<PdfFile> PdfInvoiceRepository => _pdfRepository.Value;
    public IBlobStorageRepository<CsvFile> CsvInvoiceRepository => _csvRepository.Value;

    ... // ctor etc.

    public void Commit()
    {
        try
        {
            TransactionScope?.Complete();
            CommitBlobUploads();
        }
        finally
        {
            ResetTransactionScope(true);
        }
    }

    private void ResetTransactionScope(bool dispose)
    {
        if (dispose)
        {
            TransactionScope?.Dispose();
        }

        TransactionScope = _transactionScopeFactory.CreateReadCommitted();
    }

    private void CommitBlobUploads()
    {
        PdfInvoiceRepository.Commit();
        CsvInvoiceRepository.Commit();
    }

    private void RollbackBlobUploads()
    {
        PdfInvoiceRepository.Rollback();
        CsvInvoiceRepository.Rollback();
    }
}

The primary bits to notice are the methods CommitBlobUploads and RollbackBlobUploads. These make use of the Commit and Rollback methods on BlobStorageRepository that utilise our special UploadFileCommand class:

public abstract class BlobStorageRepository<T> : IBlobStorageRepository<T> where T : IBlobFile
{
    protected readonly BlobContainerClient Client;
    protected readonly ILogger Logger;
    
    protected readonly Queue<ICommand<T>> BlobChanges = new();

    public BlobStorageRepository(
        IBlobStorageContainerFactory containerFactory, 
        IBlobStorageSettings settings,
        ILogger logger)
    {
        Client = containerFactory.CreateIfNotExists(settings);
        Logger = logger;
    }

    public Task<T> Upload(T blob)
    {
        var uploadCommand = new UploadFileCommand<T>(this, blob);

        // Use the queue to keep track of all the files we've uploaded using this repository
        BlobChanges.Enqueue(uploadCommand);
        
        return uploadCommand.Execute();
    }

    // If we want to roll back, undo one command at a time from the queue
    public async Task Rollback()
    {
        while (BlobChanges.Count > 0)
        {
            await BlobChanges.Dequeue().Undo();
        }
    }

    // If we're happy enough to commit, simply clear the queue (the uploads are already done)
    public void Commit() => BlobChanges.Clear();

    internal virtual async Task<T> UploadFile(T file)
    {
        ArgumentNullException.ThrowIfNull(file.Contents);

        var blobClient = Client.GetBlobClient(file.FullFilePath);
        await blobClient.UploadAsync(file.Contents, overwrite: true);
        file.BlobUri = blobClient.Uri.AbsoluteUri;

        Logger.LogBlobFileUpload(file.Name, file.Path);

        return file;
    }

    internal virtual async Task<T> DeleteFile(T file)
    {
        var blobClient = Client.GetBlobClient(file.FullFilePath);
        await blobClient.DeleteIfExistsAsync();
        file.BlobUri = null;

        Logger.LogBlobFileDeletion(file.Name, file.Path);

        return file;
    }
}

The above code is the final piece of the puzzle. In each of our BLOB storage repositories we keep track of a queue of commands, each of which represents a single file being uploaded. The only method we expose publicly is Upload(T blob), in order to ensure that calling code must take the path that maintains the queue, rather than calling UploadFile or DeleteFile directly (a slight oddity due to the way I’ve set it up that requires the repo passing itself to the UploadFileCommand… but hey, it works 🤷‍♂️).

So that’s it 🚀 Not the most essential code in the world and perhaps it’s a classic case of over-engineering, but hopefully it’s useful or interesting to someone!

If you’ve got any pointers, questions or you’d have gone about it differently, let me know in the comments 😊

The Background#

The Code#

The Background

The Code