John McCormack DBA

SQL Server Databases and AWS Cloud Solutions

SQL Server Threadpool Waits

By Leave a Comment

Threadpool Waits

I encountered the following error this week:

Event Description: SQL Server failed with error code 0xc0000000 to spawn a thread to process a new login or connection. Check the SQL Server error log and the Windows event logs for information about possible related problems.

This SQL Server hosts the databases for a few utility apps and has always ticked along without any concerns. However, more and more databases had been added over time and most recently Altiris was added and this is when the problems began.

Searching online, I came across this question and answer in which Jonathan Kehayias suggested the problem could be caused by threadpool starvation. It doesn’t seem to be a common problem so I had to look into to it to find out more.

In his first answer, Jonathan suggested that scaling out or splitting the workload across more SQL servers would work better than scaling up because the max number of worker threads increase would not be sufficient by scaling up. I looked into the server but realised it was had a very low spec of only 2 CPUS and 4GB memory to support 20 databases of varying sizes so in this case, maybe scaling up would help. Had I read on, I would have seen that Jonathan didn’t expect the guy’s server to have such low spec (1041 databases 1CPU 2GB RAM) and he also advised scaling up.

Measuring the threadpool waits

Querying the DMV sys.dm_os_wait_stats returns a count of waiting tasks since the last restart. I ran the query below and found that this system had experienced 18,000 threadpool waits per day since its last restart. 300,000 in total.


select *
from sys.dm_os_wait_stats
where wait_type = 'THREADPOOL'

As the stats are cumulative, I wanted to know how often the threadpool waits were happening right now. I set up a sql agent job to collect the waiting_tasks_count from sys.dm_os_wait_stats along with the date of collection and scheduled this to run every minute. I found that I was receiving threadpool waits at a rate of 822 per hour, roughly 14 per minute.

How to fix

Fortunately for me, this box was a virtual machine (VM) and as such, adding CPU and memory was a fairly seamless process. I doubled the CPUs and increased memory to 16GB to give SQL Server a fighting chance. I could go further if needed.

Once this was added, I continued to monitor the count of threadpool waits every minute and found they had reduced from 822 per hour to <1 per hour.

Summary

To summarise, although I’m sure it’s not always the answer, I found that beefing up a very low spec server had a dramatic improvement on threadpool waits. I should add this was a 64bit SQL Server running 2008 R2.

Other resources

Whilst deciding what to do and getting more information on threadpool starvation, I read a number of other resources which are listed below.

https://blogs.msdn.microsoft.com/psssql/2009/11/24/doctor-this-sql-server-appears-to-be-sick/ (Part 6)

http://sqlblog.com/blogs/jonathan_kehayias/archive/2009/11/12/the-benefits-of-attending-pass-realized.aspx 

https://www.brentozar.com/archive/2014/05/connections-slow-sql-server-threadpool/

As always, your comments and suggestions for improvement are welcome.

Disk space monitoring with SQL Server

By Leave a Comment

In this post, I discuss 3 methods (of varying complexity) to carry out some disk space monitoring with SQL server. This can be ad-hoc or you can build it into SQL agent jobs to run on a schedule. Now the first thing I’d like to say is this isn’t a replacement for good monitoring software, it’s more of a compliment to monitoring software. It’s a double check in case something goes wrong with your monitoring. Having some handy scripts is useful as well, because it lets you find out info on an ad-hoc basis, rather than relying on alerts or checking manually. [Read more…]

Compressing backups with Litespeed for SQL Server

By Leave a Comment

I recently looked at an issue where free space on a backup drive was running low. One option was to increase the storage. Another was to look at the composition of the backups and see if I could try compressing backups with Litespeed for SQL Server.

2 particular backups stood out. Database1 was 93GB and Database2 was 297GB. Incidently, this was only half of the total size as we stripe across 2 drives so the total size was 186GB and 594GB.

Example backup script for Litespeed.

exec master.dbo.xp_backup_database
@database = 'Database1',
@filename = 'D:\Backup\Database1\Database1_1.BKP',
@filename = 'G:\Backup\Database1\Database1_2.BKP',
@backupname = Database1 backup',
@desc = 'Backup of Database1',
@encryptionkey = 'LiteSp1234567',
@cryptlevel = 8,
@init = 1,
@logging = 0,
@throttle = 99,
@comment = '',
@with = 'SKIP',
@with = 'STATS = 5',
@compressionlevel = 1

For the purposes of this post, the line we are most interested in is @compressionlevel = 1. By increasing @compressionlevel on a scale from one to eight, we can reduce the size of our backups. One thing to note is that the higher the level of compression, the higher the CPU utilisation will be. More highly compressed backups will usually take more time. It is worth testing the backups using incremental increases in @compressionlevel to find the one that best suits your server.

LiteSpeed for SQL Server 7.5 allows a compression level of 1 – 8.
1 = Medium Compression
2 = Medium High Compression
3-6 = High Compression
7-8 = Extreme Compression.
(Previous versions allowed 9-11 but these are now classed as Level 8)

It is worth testing the backups using incremental increase in @compressionlevel to find the one that best suits your server.

My results:

Compressing backups with Litespeed for SQL Server

In the end, I opted for @compressionLevel = 5 for Database1 and @CompressionLevel = 3 for Database2. The increased backup time to use @compressionlevel=5 for Database1 is justifed, given the 64GB reduction in backup size. For Database 2 backup, there was too large a jump in processing time between levels 3 and 5. If we had no option, we could live with it but the original reduction in size by using @CompressionLevel = 3 was sufficient.

Side note: As a side benefit I wasn’t considering, our backup team advised the tape backup duration reduced by 1.5 hours so there was a positive knock on effect for another team too.

Source: http://documents.software.dell.com/litespeed-for-sql-server/7.5/netvault-litespeed-for-sql-server-user-guide/back-up-databases/compression-levels

How does the SQL Server engine process my select query

By Leave a Comment

This is a short post which aims to answer the question “How does the SQL Server engine process my select query”?

T-SQfwghsoL is the dialect of SQL used with SQL Server and is a declarative language. A declarative language takes the user’s request (or query) and works out the best way how to return the data. This differs from an imperative language in which the programmer tells the program how it should process the statement. With this is mind, it should be fairly easy with some simple syntax just to tell SQL Server what we want and get the results back.

An example in English could be “Bring me all my black shoes from the cupboard”. In SQL (and with a bit of stretch of the imagination), this would be written as:


SELECT item FROM dbo.Cupboard
WHERE item = 'shoes'
AND colour = 'black';

Despite the way we write the query, the SQL Server engine has other ideas about the processing order. The table below shows the order in English versus the order in SQL.

ENGLISH SQL SERVER
SELECT FROM
FROM WHERE
WHERE SELECT

So conversely, if we were to say this in English, it would be something like “From the cupboard, find all the black shoes and bring them to me”.

Why is it done this way?

First, I should point out we will be discussing 6 main clauses (but there are more). They are listed below in logical order with a description. If you read all 6 descriptions, it should be fairly clear why SQL Server processes the select query in this order. e.g. You cannot select something if the place it supposed to be (a table) does not exist. So we go to the table first.

THE MAIN CLAUSES
(In logical order)		  DESCRIPTION
FROM The FROM clause is executed first. This also includes JOIN which is a way of combining data from multiple tables into one result set. Thinking about this logically, the table(s) being queried must be identified first or we would have nothing to SELECT from.
WHERE A clause such as WHERE item = 'shoes' means that rows are filtered. Rows are not returned if the item does not equal shoes. Remember you must refer to the column names rather than an alias you have given as the WHERE clause is evaluated before the SELECT.
GROUP BY You can retrieve a set of all distinct row values based on what is included in the SELECT. e.g. 
SELECT
Item,
Colour
FROM [TestDB].[dbo].[Cupboard]
GROUP BY Item,Colour
 This can also be aggregated. In this example, I show how many times these combinations of values are repeated throughout the table by using COUNT(*).
SELECT
Item,
Colour,
Count(*) AS [Count]
FROM [TestDB].[dbo].[Cupboard]
GROUP BY Item,Colour
HAVING The HAVING clause is similar to WHERE in that it filters the data based on a predicate. The main difference is that it is evaluated AFTER the data has been grouped (and so is evaluated per group) unlike a WHERE clause which you can see is evaluated before a GROUP BY.

If we enhance the example above to include a HAVING clause (say for groups with a count > 1), the syntax would be:

SELECT
Item,
Colour,
COUNT(*) as [Count]
FROM [TestDB].[dbo].[Cupboard]
GROUP BY Item,Colour
HAVING COUNT(*) > 1
SELECT Whilst it is written at the beginning of the T-SQL query, the SELECT clause is evaluated near the end. Remember to include a DISTINCT after your SELECT clause if you want to ensure you return no duplicates.

At this stage, you are returning a relational result. You have not guaranteed the order in which you want to see the result set. This means the same query could return the same results in a different order time and time again.
ORDER BY The ORDER BY clause allows you to present the data in an order that you deem suitable. e.g.ORDER BY [Count] Desc

Notice here that because this is evaluated last and unlike the WHERE clause, you can refer to column Aliases because they have now been created.

Troubleshooting Transactional Replication in SQL Server

By Leave a Comment

SQL Server ReplicationThis might make me the odd one out but I actually really like replication. It took me a while to get comfortable with it but when I did and when I learned how to troubleshoot transactional replication confidently, I became a fan. Since I exclusively use transactional replication and not snapshot replication or merge replication, this post is only about transactional replication and in particular, how to troubleshoot transactional replication errors.

In the production system I work on, replication is highly reliable and rarely if ever causes the DBA’s headaches. It can be less so in our plethora of dev and qa boxes, probably down to rate of change in these environments with regular refreshes. Due to this, I’ve had to fix it many times. As I explain how I troubleshoot replication errors, I assume you know the basics of how replication works. If you don’t, a really good place to start is books online. It describes how replication uses a publishing metaphor and describes all the component parts in detail.

[Read more…]