Upload file without waiting for it?

I'm guessing this is more of a question around async and await, but part of my application I have to take an external link, and basically upload it to my own CDN as the links expire after a set time.

The uploading of files, remote files to be specific is rather slow so I would like to background the task and not have to worry about how long its taken, as long as I have a URL to it.

I created a storage client like:

public class SpacesStorageClient : IStorageClient
{
    private readonly AmazonS3Client _s3Client;
    private readonly string _bucketName;
    private readonly string _cdnUrl;

    public SpacesStorageClient(AmazonS3Client s3Client, string bucketName, string cdnUrl)
    {
        _s3Client = s3Client;
        _bucketName = bucketName;
        _cdnUrl = cdnUrl;
    }

    public async Task<string> UploadRemoteFileAsync(string url, string directory, string fileName, bool publicRead)
    {
        var stream = await HttpUtilities.GetStreamFromUrlAsync(url);
        return await UploadStreamAsync(stream, directory, fileName, publicRead);
    }

    public async Task<string> UploadStreamAsync(Stream stream, string directory, string fileName, bool publicRead)
    {
        var transferUtility = new TransferUtility(_s3Client);
            
        var uploadRequest = new TransferUtilityUploadRequest
        {
            BucketName = _bucketName,
            Key = $"{directory}/{fileName}",
            InputStream = stream,
            CannedACL = publicRead ? S3CannedACL.PublicRead : S3CannedACL.Private
        };
            
        await transferUtility.UploadAsync(uploadRequest);
        return $"{_cdnUrl}/{directory}/{fileName}";
    }

    public async Task<string> UploadByteArrayAsync(byte[] bytes, string directory, string fileName, bool publicRead)
    {
        var stream = new MemoryStream(bytes);
        return await UploadStreamAsync(stream, directory, fileName, publicRead);
    }
}

But the uploading of files takes far too long, I want to make UploadRemoteFileAsync smart and not have to wait, as my application doesn't need the new link straight away.

By understanding how it constructs CDN url's I believe this is possible but I'm stuck when it comes to the await stuff.

Is it as simple as removing await before UploadStreamAsync or inside it?

First, make the _cdnUrl property publicly accessible:

public string CdnUrl => _cdnUrl;

Don't await UploadRemoteFileAsync, and construct the url without waiting:

_storageClient.UploadRemoteFileAsync(sourceUrl, directory, cdnFileName, true);

Construct URL from property and other details:

var futureUrl = $"{_storageClient.CdnUrl}/{directory}/{cdnFileName}";

Let's start with usage scenarios and limitations. You probably want to start the upload, not know its completion status, but have the address you generate in the output of the UploadStreamAsync method.

In that case, you can remove await only for the actual file download method transferUtility.UploadAsync(uploadRequest) and still return the result with the address from the UploadStreamAsync method to leave the address assignment logic inside the class:

public class SpacesStorageClient : IStorageClient
{
    private readonly AmazonS3Client _s3Client;
    private readonly string _bucketName;
    private readonly string _cdnUrl;

    public SpacesStorageClient(AmazonS3Client s3Client, string bucketName, string cdnUrl)
    {
        _s3Client = s3Client;
        _bucketName = bucketName;
        _cdnUrl = cdnUrl;
    }

    public async Task<string> UploadRemoteFileAsync(string url, string directory, string fileName, bool publicRead)
    {
        var stream = await HttpUtilities.GetStreamFromUrlAsync(url);
        // we use await here but inside method UploadStreamAsync we do not
        // wait any operation to complete, so there will be no waiting actually
        return await UploadStreamAsync(stream, directory, fileName, publicRead);
    }

    public async Task<string> UploadStreamAsync(Stream stream, string directory, string fileName, bool publicRead)
    {
        var transferUtility = new TransferUtility(_s3Client);
            
        var uploadRequest = new TransferUtilityUploadRequest
        {
            BucketName = _bucketName,
            Key = $"{directory}/{fileName}",
            InputStream = stream,
            CannedACL = publicRead ? S3CannedACL.PublicRead : S3CannedACL.Private
        };

        // we run Uploading but don't wait the end of operation and go to the next line of code    
        transferUtility.UploadAsync(uploadRequest);
        return $"{_cdnUrl}/{directory}/{fileName}";
    }

    public async Task<string> UploadByteArrayAsync(byte[] bytes, string directory, string fileName, bool publicRead)
    {
        var stream = new MemoryStream(bytes);
        // we use await here but inside method UploadStreamAsync we do not
        // wait any operation to complete, so there will be no waiting actually
        return await UploadStreamAsync(stream, directory, fileName, publicRead);
    }
}

Also you can remove the address return logic and remove the use of async/await in the code. So you will manipulate the tasks maually. In this case the user will get back the upload task, for which he can wait to finish via await if he wants or discard this task if the result is not important.

public class SpacesStorageClient : IStorageClient
{
    private readonly AmazonS3Client _s3Client;
    private readonly string _bucketName;
    private readonly string _cdnUrl;

    public SpacesStorageClient(AmazonS3Client s3Client, string bucketName, string cdnUrl)
    {
        _s3Client = s3Client;
        _bucketName = bucketName;
        _cdnUrl = cdnUrl;
    }

    public Task UploadRemoteFileAsync(string url, string directory, string fileName, bool publicRead)
    {
        var stream = await HttpUtilities.GetStreamFromUrlAsync(url);
        return UploadStreamAsync(stream, directory, fileName, publicRead);
    }

    public Task UploadStreamAsync(Stream stream, string directory, string fileName, bool publicRead)
    {
        var transferUtility = new TransferUtility(_s3Client);
            
        var uploadRequest = new TransferUtilityUploadRequest
        {
            BucketName = _bucketName,
            Key = $"{directory}/{fileName}",
            InputStream = stream,
            CannedACL = publicRead ? S3CannedACL.PublicRead : S3CannedACL.Private
        };
            
        return transferUtility.UploadAsync(uploadRequest);
    }

    public async Task UploadByteArrayAsync(byte[] bytes, string directory, string fileName, bool publicRead)
    {
        var stream = new MemoryStream(bytes);
        return UploadStreamAsync(stream, directory, fileName, publicRead);
    }
}

Remember that in this case you need to change the contract for the interface IStorageClient too. And you still have an access to uploading task and can work with it:

SpacesStorageClient client = GetSpacesStorageClient();
var uploadingTask = client.UploadRemoteFileAsync(someUrl, someDirectory, someFileName, publicRead);
// wait when uploading is finished
await uploadingTask;
// wait when uploading is finished with blocking
uploadingTask.Wait();

// check if uploading is failed 
if (uploadingTask.Status == TaskStatus.Faulted)
{
    // The task completed with an exception
    Exception ex = uploadingTask.Exception;
    // Handle the exception here
}