How To Decrease ABP's Azure Costs by An Order of Magnitude

Lee Richardson -

As I may have mentioned once or twice, ABP Framework is an excellent way to jump start ASP.NET web projects.  It can propel you to production faster in the short term while improving maintainability and scalability and setting you up for success in the long term.

However, if you're deploying to Azure, you may discover that the out of the box settings are optimized more for convenience, security, and an on-premise deployment – they are definitely not optimized to reduce Azure spend.

In this post I'll share a couple minor tweaks you can implement to reduce your ABP Azure Costs by an order of magnitude.  Even if you don't use ABP, this article will help you understand how to minimize spend generally with the Azure SQL Serverless SKU.

Settings.BurnMoney = true

If you deploy ABP Framework as an App Service and an Azure SQL serverless database you're liable to see this after a few weeks:

In just 16 days, this deployment racked up $151 dollars of cost with 88% of that cost from the database alone.  That's $9.41 per day, $8.37 of which is for the database!  That's extremely high for a dev environment that contains virtually no data and was barely touched.

That wasn't for especially expensive Azure resources either.  For instance, I picked the lowest cost database SKU I could find: a General Purpose Serverless database with every setting cranked to the lowest cost position.

That looked promising when I set it up, especially that $4.78 / month estimate on the right.

MISLEADING AZURE UI ALERT: The estimate cost box when selecting a SKU is unhelpful at best.  To begin with my actual storage costs were closer to $32 per month not $4.78.  Worse, it mentions but doesn't include an estimate of compute costs.  As it turns out, my storage costs were virtually insignificant compared to my actual ~$259 / month of compute costs!

The Hidden Cost: SQL Compute

Serverless with auto-pause is supposed to be the perfect SKU for a low utilized deployment like a dev environment.  It should also be good for a production internal site that won't be used much on weekends and after-hours.

The general idea of serverless SQL is that if the database isn't used for an hour (unfortunately 1 hour is the minimum), then the database will pause itself to reduce compute costs.

However, auto pause only works if there are literally zero requests hitting the database for an entire hour.  A single request, even at minute 59 will restart the timer for another hour.

Diagnosing why your database doesn't auto-pause can be challenging.  One way you can tell if your app has successfully stopped hitting the database when not in use by going to the Database -> Monitoring -> Metrics panel and pulling up "App CPU Billed".

The "CPU Billed" for my ABP Framework with nearly out of the box settings my looked like this:

While my CPU percentage looked like this:

Translation: my CPU Utilized was near zero while CPU billed was pegged.  That meant something was frequently and very lightly hitting the database.  To track the culprit down I turned to a familiar tool.

Running SQL server Query Analyzer in my local environment was the best way I found to determine what's going on.  

With that open it was clear that every 3-5 seconds ABP was polling the AbpBackgroundJobs table.

SELECT TOP ...
FROM [AbpBackgroundJobs] AS [a]
WHERE ...
ORDER BY ...

Culprit #1: Background Jobs

Background Jobs are a wonderful ABP feature that allow you to offload run long running jobs like e-mailing or report creation to a queue for execution on a dedicated background thread.  They have a lot of great features I could extol, but it's not strictly relevant, so I'll just encourage you to skim over the documentation.

On my project I'm not using background jobs yet so I just disabled them per the documentation like this:

private void ConfigureBackgroundJobs()
{
    Configure<AbpBackgroundJobOptions>(options =>
    {
        // background jobs constantly poll the DB and run up Azure costs, so disable until we need this feature
        options.IsJobExecutionEnabled = false;
    });
}

If you wanted to decrease the polling period you could alternately do something like this:

Configure<AbpBackgroundJobWorkerOptions>(options =>
{
    // background jobs normally poll the database every 5 seconds, this
    //    updates it to every 6 hours
    options.JobPollPeriod = 21_600_000;
});#

Unfortunately, after I deployed that it didn't exactly solve all the cost problems.  After I let it run for several days, the costs decreased, but in odd, chunky, ways:

Here you can see it was now able to pause correctly sometimes, but after a hit to the site it would turn back on for days at a time.

Turning back to SQL Server Profiler, and running it for a long time, I eventually saw this:

The following SQL was hitting the database at 60 minute intervals, right when my database was about to take a well deserved nap.

SELECT TOP ...
FROM [OpenIddictAuthorizations] AS [o]
LEFT JOIN (
    SELECT ...
    FROM [OpenIddictTokens] AS [oO]
    WHERE ...
)

Culprit #2 OpenIdDict

Even though I disabled general purpose background workers, this OpenIddict background job was still running.  Digging into it further, it turned out the offending class is TokenCleanupService, which has the following description:

// Note: this background task is responsible of automatically removing orphaned tokens/authorizations
// (i.e tokens that are no longer valid and ad-hoc authorizations that have no valid tokens associated).
// Import: since tokens associated to ad-hoc authorizations are not removed as part of the same operation,
// the tokens MUST be deleted before removing the ad-hoc authorizations that no longer have any token.

The class gets run hourly by TokenCleanupBackgroundWorker, which is registered by the AbpOpenIddictDomainModule module, which in turn is a dependency of the module [MyProject]DomainModule that was generated from the starter ABP template.  

Removing the AbpOpenIddictDomainModule dependency or disabling the job seems like a poor idea, since most projects, mine included, require oAuth authentication.

However, by digging through the code one can see how to turn down the frequency.  Because of ABP's excellent customizability it's actually quite easy.  Just add this inside of your main module:

public override void PreConfigureServices(ServiceConfigurationContext context)
    ...
    Configure<TokenCleanupOptions>(options =>
    {
        // The default OpenIdDict cleanup peariod is 3,600,000 ms aka 1 hr, however, this
        // is just short enough to disable the SQL Auto-pause delay and cause it to never shut down.
        // This changes it to every 6 hours or 4 times a day
        options.CleanupPeriod = 21_600_000;
    });

And now, finally, auto-pause started working as expected.  It fired back up 4 times a day to run the cleanup job:

To reduce costs further I tried reducing CleanupPeriod to 86_400_000, or once per day.  That resulted in prices ranging from $1.33 all the way down to $0.50 for the database under light to no usage!

And for comparison purposes here's the cost of the deployment after running it for the same time period as earlier, but with the two configuration tweaks:

That's an overall reduction from $151 down to $21.  The majority of the savings were in the database costs, which went from $134 down to $7!  

Summary

Azure SQL serverless can be a great, inexpensive option for certain usage scenarios like dev environments or apps not used on weekends and evenings.  If you've got one of those scenarios and you're using ABP though, you will need to adjust your background job and openiddict settings to maximize your cost savings.  I hope this post helps you to massively reduce your Azure costs.