the lazy hacker blog

Documenting dbic++

2010-08-18

I had a look at naturaldocs and decided to document dbic++. I don’t think it was a brilliant idea given I started it with a glass of scotch by my side.

I may have to proof read in the morning given its almost 11pm here. But for rest of you night owls, here it is – dbic++ docs

An object mapper that's uber fast and lean.

2010-08-12

@shanehanna recently released a first cut of the Swift Object Mapper to rubygems. Swift has come along a fair bit since we started hacking on it as a hobby to see if we can put to rest the criticism that ruby is slow and takes up too much memory.

I ran some benchmarks this morning and found it to be 4x fast on average compared to ActiveRecord and DataMapper. It uses considerably less memory (90% less) as well. If you are inserting a lot of records, you should consider using the Swift::Adapter#write method, as it can lead to tremendous speedup (~50-80x) and memory savings.

Why reinvent the wheel you ask ? Take a look at the numbers below and you will understand. ActiveRecord and DataMapper come with a lot of features and hence some pretty heavy baggage. We realized that most of the time, all we used them for was mapping properties to fields in the database and some basic validations. We realized that we could do this in a much more simpler fashion.

More coming soon …

Swift is not as feature rich as ActiveRecord or DataMapper, it was meant to be a simple and thin abstraction. We’re planning on working on swift-more soon which would have a ton of goodies including validations, associations etc.

Also planned are adapters for SphinxSearch and Mongo. SphinxSearch 1.10 allows you to store fields directly in addition to making them searchable. We plan to either use the real-time indexing feature or xmlpipe2 to persist objects in sphinx directly.

The benchmarks

The bechmark code for each ORM,

Creates 10,000 records at the start.
Selects all the records from the database and goes through the results once.
Selects all the records and updates the fields once for each record.
GC is disabled to measure the actual memory footprint of each test run.
ActiveRecord uses mysql2 and pg gems and Swift uses bindings to dbic++.

You can clone the repo, run the tests and see for yourself.

MySQL

benchmark       sys          user        total       rss
dm #create      0.360000     3.650000    4.010000    227.14m
dm #select      0.140000     1.880000    2.020000    128.94m
dm #update      0.710000     8.330000    9.040000    593.20m

ar #create      0.570000     5.360000    5.930000    304.77m
ar #select      0.020000     0.170000    0.190000    22.92m
ar #update      0.560000     5.710000    6.270000    322.95m

swift #create   0.310000     0.680000    0.990000    30.61m
swift #select   0.000000     0.100000    0.100000    9.32m
swift #update   0.140000     0.660000    0.800000    32.26m
swift #write    0.000000     0.070000    0.070000    6.79m

PostgreSQL

benchmark       sys          user        total       rss
dm #create      0.360000     3.560000    3.920000    245.34m
dm #select      0.110000     1.910000    2.020000    129.11m
dm #update      0.590000     7.910000    8.500000    604.77m

ar #create      1.030000     7.100000    8.130000    367.80m
ar #select      0.030000     0.320000    0.350000    38.82m
ar #update      0.800000     6.630000    7.430000    361.91m

swift #create   0.220000     0.680000    0.900000    30.60m
swift #select   0.010000     0.070000    0.080000    9.82m
swift #update   0.100000     0.890000    0.990000    32.66m
swift #write    0.000000     0.080000    0.080000    7.30m

Bulk loading data into MySQL

2010-07-26

During a casual conversation last week with a friend, we happened to talk about how you can bulk load data data in PostgreSQL without writing to temporary files via the PQputCopyData API.

The MySQL client API does not have an equivalent way of streaming data right from your program/script. So I started thinking about how I can do this in MySQL. MySQL C API does allow you to specify your own infile handlers but it seemed unnecessarily complicated.

~~A few drinkies later, I tripped on a neat solution. named pipes – et voilà!~~

So all I had to do was create a named pipe, fire up a thread and stream the data to the pipe in that thread and let the mysql client library read from it. No temporary files, no cleanups (except for unlinking the FIFO file).

In principle this is same as

mkfifo -m0666 /tmp/somefile.fifo
zcat users.gz > /tmp/somefile.fifo &
mysql -uroot mytestdb -e "load data local infile '/tmp/somefile.fifo' into table users;"

UPDATE Although named pipes worked like a charm it started to cause issues with threaded applications in ruby. So I had to implement custom infile handlers for dbic++ which was a bit more code but works like a charm.

But a nice little api around this helps!

require 'etc'
require 'swift'
require 'stringio'

data = StringIO.new "sally\tsally@local\njerry\tjerry@local\n"

Swift.setup :default, Swift::DB::Mysql, user: Etc.getlogin, db: 'swift'

Swift.db do |db|
  db.execute 'drop table if exists users'
  db.execute 'create table users (id serial, name text, email text, primary key(id))'
  db.write   'users', %w{name email}, data
end

The block can stream data from anywhere, file, network etc and straight into MySQL. You just have to make sure you do it in batches of reasonable size to avoid MySQL writing too much to its binary log (WAL in case of PosgreSQL) in case you have to rollback.

Grab the latest dbic++ and swift, have fun!

Allow me to introduce dbic++

2010-07-18

As promised, I give thee dbic++, a kick ass database abstraction. Clone away!

Mind you it’s still under heavy development, so don’t use this on any critical production code.

How fast can you go ?

2010-07-16

I started writing a database abstraction library in C++ (libdbic++) to see if I can write something simple and fast that I can use at work. I was surprised at how fast it did manage to go.

I also wrote ruby bindings (swift) with help from @shanehanna. This makes it easier to start writing apps in ruby and then switch to C++ if performance becomes a bigger priority. A single API that makes it easy to transition from ruby to C++ was the goal.

The performance numbers for retrieving 250,000 records of approximately 64 bytes each was quite encouraging. We intend to hack away at this until we have something nice and polished.

mysql
=====
                                     user     system      total        real
 ruby: do                        3.350000   0.110000   3.460000 (  4.431064)
 ruby: swift                     1.080000   0.110000   1.190000 (  1.905136)
 c: libmysqlclient               0.000000   0.000000   0.230000 (  0.933122)
 c++: libdbic++                  0.000000   0.010000   0.510000 (  1.163342)
 ruby: mysql2                    2.150000   0.040000   2.190000 (  3.155557)
 c++: libmysql++                 0.000000   0.000000   0.480000 (  1.437604)

postgresql
==========
                                     user     system      total        real
 ruby: do                        3.470000   0.130000   3.600000 (  4.583538)
 ruby: swift                     1.300000   0.070000   1.370000 (  2.010762)
 ruby: libpq                     1.800000   0.120000   1.920000 (  2.815204)
 c++: libdbic++                  0.000000   0.000000   0.560000 (  1.336096)
 c: libpq                        0.010000   0.000000   0.510000 (  1.085639)

A clean and simple database api

2010-07-15

Recently, I started looking for a simple C/C++ database abstraction layer with a nice and simple API. The best I came across was libdbi.

I wanted something that did the following,

PostgreSQL and MySQL support – because those are the 2 main databases we use at work.
Transaction and Savepoint support.
Support for non-blocking async calls.
Support prepared statements.
Do all this in a fast and simple fashion.

Unfortunately libdbi did not have some features such as support for prepared statements and support for asynchronous access. Coming from years of Perl development, I expected something along the lines of DBI and was disappointed.

I wanted to try writing something over the next weekend to see if I could come up with something simple and fast that works with PostgreSQL and MySQL.

After a few weeks of toying around, I ended up with dbic++. What started as a learning exercise ended up becoming a pet project that started to look better than many of the existing open source libraries. It also ended up outperforming most of them.

I will be releasing the code sometime next week – on github, most likely. I’ll post a link. Meanwhile here’s an example that provides a brief glimpse of the API.

#include "dbic++.h"
#include <unistd.h>

using namespace std;
using namespace dbi;

int main() {

    // Handle h ("driver", "user", "password", "database", "host", "port");
    Handle h ("postgresql", getlogin(), "", "dbicpp");

    Statement st (h, "SELECT id, name, email FROM users WHERE id >= ? AND id < ?");

    // bind and execute the statement.
    st % 1L, 10L;
    st.execute();

    ResultRow r;
    while (r = st.fetchRow())
        cout << r.join("\t") << endl;

    // or you can do
    st.rewind();
    ResultRowHash rh; 
    while (rh = st.fetchRowHash())
        cout << rh["id"]    << "\t"
             << rh["name"]  << "\t"
             << rh["email"] << endl;

    st.finish();
}