Datamangling

LIMIT 1 and performance in a Postgres query

2014-01-17T00:00:00-08:00

I’ve just been debugging a slow-running report, and reduced the runtime from over 3 hours to under 5 minutes. The problem turned out to be the impact of LIMIT 1 on a simple SQL query on a Postgres DB.

The report is fairly straightforward, and the queries it executes are also pretty simple, so it was a surprise to see how slowly it ran in production.

The script is written in Ruby, running against one of our Rails applications. Instrumenting the script showed that one particular query, which only used one table, was often taking several seconds. It wasn’t obvious to me why it should be so slow. The table has several million rows, but sensible indexes are in place, and we don’t have any problems in production. I restored a backup of the DB to a isolated box, and started to experiment.

I took the generated SQL of the query, and looked at the query plan and performance. EXPLAIN ANALYZE is your friend here, as you get both the query plan and the actual runtime of the query:


EXPLAIN ANALYZE SELECT field1 FROM table WHERE field2 = value ORDER BY field3 DESC LIMIT 1;

QUERY PLAN
--------------------
  Limit  (cost=0.00..524.25 rows=1 width=34) (actual time=5572.027..5572.028 rows=1 loops=1)
  ->  Index Scan Backward using index_table_on_field3 on table  (cost=0.00..569861.98 rows=1087 width=34) (actual time=5572.018..5572.018 rows=1 loops=1)
Filter: (field2 = value)
  Total runtime: 5572.056 ms

A runtime of over five seconds. How many rows does are we working with?


SELECT COUNT(*) FROM table WHERE field2 = value;

  count
-------
  1020

5.5 seconds to pick a single row from just 1020 records? Something isn’t right. How long would it take to get ALL THE RECORDS? I dropped the LIMIT 1 from the query:


EXPLAIN ANALYZE SELECT field1 FROM table WHERE field2 = value ORDER BY field3 DESC;

QUERY PLAN
--------------------
Sort  (cost=1080.91..1083.63 rows=1087 width=34) (actual time=46.318..47.194 rows=1020 loops=1)
  Sort Key: field3
  Sort Method: quicksort  Memory: 104kB
  ->  Index Scan using index_table_on_field2 on table  (cost=0.00..1026.09 rows=1087 width=34) (actual time=0.245..44.305 rows=1020 loops=1)
Index Cond: (field2 = value)
  Total runtime: 48.199 ms

WTF? 48ms to return all the records, but 5.5s for just one? Also note that the query planner is now using the index on field2 (i.e. 1020 rows) instead of the index on field3 (several million rows). I hit the internets, and found that adding a LIMIT can often confuse the query planner. This post suggested adding conditions to the ORDER BY. I added field1 to the ordering and the results were startling:


EXPLAIN ANALYZE SELECT field1 FROM table WHERE field2 = value ORDER BY field3 DESC, field1 LIMIT 1;

QUERY PLAN
--------------------
Limit  (cost=1031.53..1031.53 rows=1 width=34) (actual time=3.856..3.857 rows=1 loops=1)
->  Sort  (cost=1031.53..1034.24 rows=1087 width=34) (actual time=3.853..3.853 rows=1 loops=1)
  Sort Key: field3, field1
  Sort Method: top-N heapsort  Memory: 25kB
  ->  Index Scan using index_table_on_field2 on table  (cost=0.00..1026.09 rows=1087 width=34) (actual time=0.028..2.667 rows=1020 loops=1)
Index Cond: (field2 = value)
  Total runtime: 3.890 ms

So we’ve gone from 5.5 seconds to just 3.9ms, by adding an irrelevant condition to the ORDER clause. Adding .order(:field1) to my Rails script got that added to the generated SQL, and the run time of the whole report reduced from over 3 hours to under 5 minutes.

Location, Location, Location at ACCU 2013

2013-04-16T00:00:00-07:00

Slides from my “Location, Location, Location” talk at ACCU2013 in Bristol on 12 April 2013.

The code can be found here.

Location Location Location at ACCU 2013

View more presentations from Gavin Heavyside

DevOps and Infrastructure-As-Code at ACCU 2012

2012-05-05T00:00:00-07:00

Slides from my “DevOps and Infrastructure-As-Code” talk atj ACCU2012 in Oxford on 27th April

I presented a 90-minute overview of infrastructure-as-code, chef, Vagrant, and zero-downtime deployment techniques.

DevOps at ACCU 2012

View more presentations from Gavin Heavyside

ACCU 2011 - Non-Relational Databases

2011-04-20T00:00:00-07:00

These are the slides from my talk last Saturday at the ACCU2011 Conference in Oxford.

I presented a 90-minute whirlwind tour of non-relational database families (trying my best to avoid saying ‘NoSQL’), and went into a little more detail for one in each of the column, graph, document and key-value families.

These slides were chosen by Slideshare to be showcased on the front page of the Technology category

Non-Relational Databases at ACCU2011

View more presentations from Gavin Heavyside

Running a Jekyll blog on Heroku

2010-07-14T00:00:00-07:00

This blog used to be hosted on the Jekyll-powered Github Pages, and I was using the Custom Domains feature to have it appear under with the DataMangling domain name. I’ve switched to
hosting it with Heroku so that I can save myself $84 per year.

Github only let you use Custom Domains if you’ve got a paid account. I had a micro (7$/month) plan, initially because I wanted to keep a couple of my repositories private, but I don’t need to do that anymore. I’m on a bit of an economy drive at the moment, so the monthly payment to Github has been a casualty of my belt-tightening.

The only tangible consequence is I can no longer alias my own domain name to the blog. This is a bit annoying after I’d gone to the trouble of finding a dotcom domain that had the word data in it. The solution is to host the blog on Heroku, as the free tier of service is more than sufficient, and I can use my own domain name.

The only question was how to get Heroku to serve the Jekyll blog properly?

Enter Rack-Jekyll. By depending on the rack-jekyll gem and adding a config.ru for Rack awareness the transition was smooth and painless. The only steps involved were:

Add rack-jekyll to the .gemfile
Add a config.ru as described in the rack-jekyll page
Make sure the _site directory is checked into git
Create a new heroku app and add my custom domain name
Update DNS to point to Heroku
Push to Heroku

Notes on Trying to Develop for Symbian

2010-07-13T00:00:00-07:00

Aaaaaaaaaaaaaaaaaaaaaaaaaaarrrrrrrrrrrrrrrrrrrrrrrgggggggggghhhhhhhhh.

Cascalog Lightning Talk at HUGUK

2010-06-03T00:00:00-07:00

The 4th Hadoop Users Group UK meetup took place last night at the Skillsmatter Exchange.

Aaron Kimball of Cloudera talked about Sqoop, and Tim Sell of Last.fm talked about how they use Hive. Ben “Shevek” Mankin of Karmasphere gave a Lightning talk introducing Karmasphere Studio which looks great, and then I gave a quick overview of Cascalog.

I think the combination of two unfamiliar technologies (Clojure and Datalog) was probably a bit much for most people, but it was good to be able to talk about something new.

HUGUK #4 - Cascalog

View more presentations from Gavin Heavyside

ACCU 2010 session - Introduction to Hadoop

2010-05-28T00:00:00-07:00

At the ACCU2010 Conference in April I presented a short session introducing MapReduce and Apache Hadoop, into which I crammed HDFS, MapReduce, Hadoop Streaming, Pig and Hive. Slides are available on slideshare:

Introduction to Hadoop - ACCU2010

View more presentations from Gavin Heavyside

GeoTools Quickstart example in Clojure

2010-05-26T00:00:00-07:00

As part of my Clojure experiments I’m taking a look at GeoTools and the Java Topologya Suite for working with geospatial data.

Clojure has been described as “A better Java than Java”. I’m not a Java programmer, but having access to Java libraries in Clojure is very useful, and Clojure has made the interop remarkably painless so far.

The quickstart example on the GeoTools documentation pages starts with a horrid Maven incantation which I converted to a leiningen project file. The source itself once converted to Clojure looks like this:

And the leiningen project.clj:

Running either of the functions pops up a dialog box for you to choose a shapefile, which is then loaded and displayed.

The clojure version is shorter than the Java version, can be tinkered with at the REPL, and doesn’t use any Maven.

It's Good to be Lazy

2010-05-19T00:00:00-07:00

On 14th May 2010 the inaugural meeting of the Software Craftsmanship UK user group was held at the offices of Eden Development in Winchester.

The meeting kicked off with introductions, and then Enrique called Doug Bradbury and Micah Martin of 8th Light on Skype, and they talked to us about the history of the Software Craftsmanship movement.

After lightning talks, we moved on to a randori-style coding dojo in which the task was to write an algorithm to determine how many Lychrel numbers there are in the starting range 1-10000.

We used Ruby, which was known by the majority (but not all) of the attendees, and after a few false moves a recursive algorithm took shape and an answer was found. The program could have used a bit of refactoring, but it satisfied the task.

I’m currently very enthusiastic about Clojure, partly because I’ve been meaning to learn a functional language for ages and this looks like a good one, and partly because it runs on the JVM and has some interesting libraries that I want to use with Hadoop for processing and analysing big data sets at Journey Dynamics.

The next day I knocked up a Clojure solution to the problem we’d seen in the dojo. It was concise, tested, and I was fairly pleased with it. I tweeted about it, and was about to forget about it until @t_crayford said that he could do better. Before long he’d posted an improved version of the main function:

At first I was baffled, but that was mostly down to trying to read it on my iPhone after a couple of glasses of wine. With a clear head and a big screen the next day it became obvious how he had replaced my naive recursive algorithm with a much more idiomatic lazy sequence version that has better performance.

He defines a function that calculates the next number in the sequence, and creates a (infinite) lazy sequence of them using iterate. It takes 50 numbers from this sequence using rest and take and checks to see if any of them are palindromic using some. I was initially confused by the ->> macro, but it is explained here.

It’s going to take me a while to think naturally in Clojure idioms, but I think it will be worth it. Paul Graham argues that the truly serious hacker should consider learning Lisp, and I think he is absolutely right. The advantage of being able to think about solutions to problems in a different way from the dominant procedural and OO mindset can be really valuable, and I agree that even if you don’t subsequently use Lisp, having learned it will make you a better programmer.