Find your targets: the slow query log

First, make sure you have the slow query log turned on and properly configured. Here’s an example /etc/my.cnf section:

Add this to your current configuration (or edit as necessary) and restart the MySQL server. The slow query log adds no overhead and is very useful in finding what queries are taking how long on your system.

mysqlsla: slow query log analysis

mysqlsla can parse and analyze all manner of MySQL logs. In this case, let’s use it to analyze our slow query log, assuming the server has been running for a bit to collect an acceptable sample of slow queries. One day to a week should be sufficient, depending on your usage patterns.

Download, made executable, run:

You minimally need to specify the path to your slow query log and indicate the type of log you are analyzing. Additional options are available for you to specify what in the log to analyze including the specific database and the type of query used. See the documentation.

What you’ll get out is a nicely formatted informative report of the queries found in the slow query log, sorted by what took the longest. You will see how much time was spent in a locked state, how many rows were sent and examined, what users were involved, the query itself, and an abstracted form of the query with particular parameter values generalized:

Looking through this report will allow you to get a sense of what queries are taking up the most of your instance’s time (and are therefore your best points to focus on).

Analyzing target queries with the Query Profiler

I discuss the basics of the MySQL Query Profiler in this post. Let’s assume you’ve found a few queries using mysqlsla as shown above that you want to learn more about and optimize. You could wrap them in a file like so:

And run this wrapper script as:

The buffer queries give you markers in the output to jump to if you want to filter out the actual data results from your output file (since you’re probably just concerned with the profile data at this point, having a well-defined query). I’ll leave it to the reader to come up with an improved command to remove the query output from the file generated, producing a CSV for each report. (Add to the comments if you make this please )

So to zoom in on the most problematic queries we just need to:

• Enable the slow query log
• Process representative slow query log entries with mysqlsla
• Analyze most troublesome queries with the MySQL Query Profiler

Toggling Profiling on and off is a matter of setting a session variable, aptly named ‘profiling‘. Here’s how to set and read it:

Once on, subsequent queries you run will have profiling information collected about them. By default the last 15 queries’ profiling information is kept, but you can increase this up to 100 by setting profiling_history_size. To view the profiles collected so far just run SHOW PROFILES:

The queries are identified in sequence of when they were run. You can view how long they took as well as the query itself. Once you select the one you want to get information on, the simplest way to get it is a variant of the SHOW PROFILE command:

This describes every step of the process from checking the query cache to returning the result and cleaning up with the duration for each step. Awesome! You can also dive into more particular data by passing a type parameter after PROFILE, such as swap and cpu. I’ve had mixed results with those, so I’ll wait until I learn more. Info in the docs if you want to give it a try.

Note of caution: If you show the profiling information for a more complicated query you might get a LOT of output. If, for example, each row in your resultset required touching a view, you might start to get hundreds or thousands of lines. This can be a little disconcerting when it first happens. After it has stopped or you kill it, check out the information_schema.profiling table and manually pull the information you need. This helpful article provides an example (about halfway through) where aggregation functions were used on this information_schema table to provide the information you’re likely looking for.

For another example of an application of the Query Profiler, check out this post demonstrating the effectiveness of the query cache. That same article also provides an example of using the Query Profiler to show what MySQL source functions are being called as well, very interesting!

Yeah, that’s the actual URL that works. Pretty sweet.

Anyway, next you make it executable, run it, enter creds:

Then the goodies appear:

Oh, that query_cache_limit is tiny! At this point, no changes have been made to your setup. But at a glance you get helpful stats, validate that changes you’ve decided to make are in place, and get alerted to potential improvements.

First an example view that I created using the handy world database:

This is a view of the 10 cites with the lowest population in the Central Luzon region of Asia. Now let’s see what actually got stored. To get a view’s create statement you can use the MySQL Query Browser and right click the view name shown in the right under Schemata, selecting “Copy SQL to Clipboard”. Or you can run a query like:

Here’s the result for the view above:

I added the line breaks for a modicum of readablilty, the statement is stored without them. I sure don’t want to edit that version in making a variant! This example isn’t actually too bad, but for real world complicated views, it really is quite difficult to read views exported in this way. I often try to store the original create view query just to avoid this pain. But sometimes I only have the stored syntax.

I’ve created a Bash function to make this whole process less painful:

Now to get a cleaned up view you can call this function followed by the database and view names:

To get the create statements for multiple views you can use a loop. For example:

If you need some more flexibility than this I implemented it in Python after an initial fit of rage caused by trying to get various levels of nested strings properly escaped in Bash. That ended up being overly long and unnecessary, and a cooler head prevailed. But the script is here on GitHub if you want it. It creates the same file although without using sed and with some error handling.

Some examples are in order. I’ll be using the world database. The following shows country names and a language spoken therein for European countries that speak Arabic, Dutch, and/or Greek:

This returns:

But what if we want Dutch speaking countries first? We can’t use order by Language desc or asc since Dutch results are in the middle of either sorting. Instead:

This orders the results with records having Language = “Dutch” first, followed by all the others ordered by their Language value. And this is the desired result:

You can specify as many values in the order by clause as you want, each also allowing an asc or desc modifier. If you have a number of values in one field you want to specify, the field keyword can save you some typing:

Returning the customized ordering:

Since I normally only use it on my own machines for testing, I’ve found setting it up to be a fixed process. So I created this alias for such situations:

This pulls down the world database dump, unzips it, creates a DB to hold it, imports it, then removes the temporary files. You’ll be prompted twice for the password of the root MySQL user. You could also alter it to include the password if you are so inclined. Add the above alias line to ~/.bashrc, reload it, and you can have a test DB up and waiting within seconds anytime you want.

Assuming you have binary logging turned on, you first need to know where the binary logs are. The query “show binary logs;” will give you the filenames and sizes, but not the path. If you only set “log-bin” in my.cnf then the binary logs are placed in your data directory by default (find this by running “show variables like ‘datadir’;”). Otherwise they will be wherever you specified with “log-bin = /some/path”. If you passed the value via a command line option, run “ps aux | grep mysql” to view the options passed. If all else fails, run “sudo updatedb && locate mysqld-bin”.

Now how to view these logs? There is in fact a handy utility for just this purpose: mysqlbinlog. Just pass it the name of the log file and it returns the text version. So if you only had a single search you could use:

If you had a number of searches you could redirect to a file for processing at your own pace:

This gives you the ability to search across all of the statements affecting your DB, as well as across all of the data coming in.

It only covers the very basics, but then you don’t need to remember those at least, and save yourself some typing to boot. Here’s what it does:

• Set a root password. If you already have it, you’ll need to enter it for the remaining steps.
• Remove anonymous users
• Disable non-local root access
• Remove the test database and access rules related to it
• Reload privilege tables so the above changes are in effect

A quick and easy way to follow good practices.

What follows is the setup I have settled on after a good bit of trial and error. It involves a little more effort than the minimum needed, but it’s a lot safer and more informative. It makes use of the logging module, whose setup I will assume. For more information on it, check out my article on simple logging in Python. You’ll wonder how you lived without it.

After developing with it for a while, I’ve found it’s easy to forget that the MySQLdb module isn’t included in the standard library. You need to verify it’s installed in the scripts you use it in. The following should suffice (assuming you aren’t using a package that sets up your dependencies):

This lets you know where to go to find the module and exits with error.

Now you have access to the MySQLdb module. But what’s the best way to access its functions? I have found it easiest to create a function that accepts a dictionary of the needed connection information as well as the query to run. It returns the data found, handling and logging errors. After this is in place, you only need to define your DB parameter dictionary (or load it from a config file), and then wrap the call to the query running function with a minimum of error handling. Here’s the function to perform the queries:

This would be called by:

This makes it simple to run any query against any database setup using a single function. The parameters included in db_params should cover most any Linux MySQL setup. One exception is socket, as this may differ between distributions. If you aren’t sure, just run “ps aux | grep mysql” and the process shown (assuming it’s running) should include a “–socket” option specifying where the socket file is. For servers on Windows, remove socket in favor of named pipes or shared memory. Or just use the TCP/IP default via port.

If there is an error trying to connect to the server, it is logged and the script exits. The log will include the numeric MySQL error code as well as a more or less helpful error message. Check out this page for more info on MySQL error codes and message. This approach seems to work out well, since an issue in connecting means that there is likely either a problem with the database server, or our connection information is wrong. In either case, we will probably want to stop and re-write something.

If the connection is successful, it next attempts to run the query passed. If there is a problem, it raises the error to the calling function, then cleans up the database call and connection. Finally, the data are returned.

This function allows for fast access while being fairly simple and easy to follow. It has served me well. Here is a more complete example of its use:

A word of caution: One thing that might be surprising is that queries resulting in warnings will error out. MySQL warnings aren’t always that bad. They might just be notes, which are informative and not critical in nature. I suggest writing queries that don’t generate actual warnings, and turn off note displays by turning setting the sql_notes session variable off (they really just don’t seem to be often useful).

And so too are we who still need to find the extreme value among columns that can contain NULL! The problem might have been solved by MySQL releasing two new functions at the same time as the greatest()/least() changes that would perform the same operation as their old behavior, ignoring NULL values. It’s a common enough use case. But complaining won’t write queries.

The first attempt I made used a combination of least() with coalesce(). This function returns the first non-NULL value among those passed. The problem with this approach is that any of the column values per row can be NULL along with any of the others. So I would have needed to cover all possible permutations, namely the number of columns to compare across squared! This gets immediately tedious. Consider the case for just four columns:

Yuck. I nearly wrote a script that would take the columns to be compared and generate the needed query.

Luckily, fellow information wrangler Bill Karpovich suggested a much more elegant (and blissfully shorter) version:

This checks each column to be examined, and considers NULL values to be equal to ’9999-99-99′, which I feel safe in assuming is greater than any valid date I will have. This method fulfills this common need nicely, and in syntax only requires a number of statements linearly increasing with the number of columns.