I’ve dabbled with encryption several times over the past few decades, never really getting serious about it.  It started when, in college, I would see that the faculty in the CS department had these weird signature blocks appended to the their USEnet posts containing something called a public key.

This is like setting a can of lighter fluid and box of blue-tip matches in front of a 10 year-old boy — irresistible.

What I found in the later years is that using encryption for email is a lot like being one of the early adopters of the telephone.  Now that I have one, who am I going to call?  My family certainly doesn’t use encryption in their email…those that have discovered email anyway.

Co-workers aren’t likely to invest the time and effort into encryption simply because we’re all too busy with work to be playing spy-games with our de-coder rings.

And, Dorothy, we’re not in Academia anymore where, I imagine, it’s really in use.  Closeted anarchists posting semi-heretical Berkeley-esqe rants against the Proletariat and all that.

And then there’s the whole nouveau post-9/11 trend of “Guilty until Proven Innocent” thing happening.  I imagine some fedora-capped DHS agent squinting at me in a menacing fashion while I try to reason a plausible excuse for being so brazen as to need encryption for my emails in the first place…

So, at this point, let’s assume that, like me, you’re willing to whack the hornet’s nest with a stick and use encryption for your emails and that you actually have someone on the other end willing to dust-off the de-coder ring and play with you.  We’ll also assume that you know what PGP, GPG and OpenPGP actually are, and that you know how basic public-key encryption works.  (If not, leave comments to this article and I will do a future article explaining same.)

Standard Disclaimer: I am providing this tutorial as a hands-on, learn-with-me type of tutorial.  I am not an expert, nor do I pretend to claim anything other than neophyte status when it comes to encryption.  I do not advocate, support, or intend for you to use this, or any, technology as a means to intentionally bending, fracturing, or breaking laws in one or more jurisdictions.  MY only intent is to share what I’ve recently learned with you and to have some fun.

OK – that crap out of the way, let’s get started.  First, as the title implies, this set-up is for Apple Mail under OS X Lion.  The release of the OS I’m currently working on is 10.7.4.

Download and install the GPGTools utility  (Version 2012.03.18 as of this writing.)

Although this article is for Apple mail, the GPGTools utility includes support for Enigmail in Thunderbird 7.  When you launch the installation utility, you’ll be presented with a list of packages to install.  I installed all packages.

Once the install has been completed, you’ll see a little dialog box appear on your desk top telling you the installation was successful, and would you like to read the Quickstart Tutorial?  This would be a good thing to do because I am not going to walk you thought the next steps in any great detail.  This reference, however, does.  With pictures.  So… go there and follow the installation step to:

Generate a key

You will generate a public and a private key.  Anyone with whom you wish to exchange encrypted email with must also have done the same.  They’re called public and private keys for a reason.  One you share with the public and one you do not.  Key, using the nomenclature, are stored on what’s called a keyring.  There are public and private key-rings.  GPGTools refers to key-rings as  keychains - these are one and the same things.

Please note that for whichever email account you’re going to use to generate a key-pair for, that account must already exist in Apple mail.  The email address is case-sensitive so make sure you type it in exactly as it is stored in Apple Mail — otherwise, your encryption will not work.

Over the years, I have created several key-pairs for various email addresses I have had.  What’s critically important to remember is this:  write down your pass-phrase.  Also, click on the Advanced options tab, and set an expiration date (a couple of years is fine and 4-years is the current default – point being: set an expiration date) for your keys.  That way if, after a few years, you return to a previous email account address, and you’ve certainly forgotten your passphrase from lack of use, then you’ll still be able to generate a new key pair if the old one has expired.  To remove a key pair, most public key rings require you to enter your passphrase.  This is known as a conundrum.

Once you’ve created your passphrase and uploaded your key, and you can see your new key in your keychain, open Apple Mail.  Send an email to

the email address you’ve just created (I know…) and you should see two buttons appear in the lower-right corner of the header bar as shown in the image on the right.

The two buttons, as shown above, allow you to either sign or encrypt your mail message.

Signing your email is flagging the email to the recipient assuring them that it was actually you who sent the mail.  In order to sign an email, OpenPGP has to have access to your private key. (You did keep your private key private, right?)  Since you’re the only one, presumably, with access to your private key, then signing the mail guarantees to the recipient that the mail did come from you.

The recipient does not need to have your public key, nor do you need to have the recipient’s public key, to sign an email.  Think of this as the “certified mail” from the US Post Office equivalent for email.

If you have a recipient’s public key, then you may send them an encrypted email.  The recipient will need to have your public key in order to decrypt and read the email — this is why we store public keys on public key rings.

Source code

-----BEGIN PGP SIGNATURE-----
 
Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
 
iQEcBAEBAgAGBQJPtnyiAAoJEC4S4zGLhwvBFNMH/1Yoh59etAcYZpAhZ+htpd81
QzZWDxOR2PeXtPkY3GWl4vdW7GABJ9ysl8vpdErsDtXs6LEVZXag5mV6CGTDNXmm
pdozUJCgNwbHTgoIUdjinmAXLR+4pYSfALTB1S2qpxzMpykBkR7SMuPm3+0LC77/
dwnsSVx5CNtJd8cPoPjwXJ6zaStJCNK+H17MItS5kpw3MqMU35qZdNCDV6ehhA8j
FmTyFoh1TeTmuBrNECWz9z3KniG6SWVl3K21LmS8PQExeHq8qcHGBz5yK2YhoW/w
bn4PIyHaUiXKQTNhYBSd1DrCPUWJKDJ+VCKQ0L97aUPeVPQBI14jsFOgc1dwUjs=
=xfmB
-----END PGP SIGNATURE-----

That’s pretty much it — once you send a signed an encrypted mail, you can rest assured that (hopefully) your emails are safe from casually-prying eyes as they’re no longer being sent in clear-text across the ether sphere.

Here’s the raw-text (what’s sent out over the ether) of an encrypted email message:

Source code

Return-Path: <mshallop@gmail.com>
 
Received: from [192.168.0.2] (c-50-136-203-107.hsd1.ca.comcast.net. [50.136.203.107])
by mx.google.com with ESMTPS id qu6sm6794406pbc.36.2012.05.18.09.54.57
(version=TLSv1/SSLv3 cipher=OTHER);
Fri, 18 May 2012 09:54:58 -0700 (PDT)
Content-Type: multipart/encrypted; boundary="Apple-Mail=_4C5344B9-76FE-43EF-8620-073841EBF944"; protocol="application/pgp-encrypted";
Subject: test both
Mime-Version: 1.0 (Apple Message framework v1278)
X-Pgp-Agent: GPGMail 201 (a30)
From: Micheal Shallop <mshallop@gmail.com>
Date: Fri, 18 May 2012 09:54:56 -0700
Content-Transfer-Encoding: 7bit
Message-Id: <407C94BA-32A1-4930-B9F6-BBFE7900D213@gmail.com>
Content-Description: OpenPGP encrypted message
To: Micheal Shallop <mshallop@gmail.com>
X-Mailer: Apple Mail (2.1278)
 
This is an OpenPGP/MIME encrypted message (RFC 2440 and 3156)
--Apple-Mail=_4C5344B9-76FE-43EF-8620-073841EBF944
Content-Transfer-Encoding: 7bit
Content-Type: application/pgp-encrypted
Content-Description: PGP/MIME Versions Identification
 
Version: 1
 
--Apple-Mail=_4C5344B9-76FE-43EF-8620-073841EBF944
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename=encrypted.asc
Content-Type: application/octet-stream;
name=encrypted.asc
Content-Description: OpenPGP encrypted message
 
-----BEGIN PGP MESSAGE-----
Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
 
hQEMAy4S4zGLhwvBAQf/YX1vGFhG0CLd7UU79fjHd4/nIHH9DVVsi8oqsmIwBNpl
zXvDZf3+uw3B7Shk3bls1fHcUdU8LZprY+HbQVZlh7IiGZ28K67rNIHKUtwuoX2I
DpZcLdRPGn1iGi7TNRs/3Fn3fjCCT15uVAbZZRvR1G3XqUc+//3TstCMTaNtg5Rl
KZbKnnrOaIt/OAou7BFCLQgGKAAIa3m/gFHEQxxLVaAQ3JISeX4/UZ7YlXAj5SYp
juVRKfekmjpoFoM4erf0Jjaw63lSjAZWXJi1m6IY8uSkzQZRUwANMYE577mBiZhw
d1kIBJfBlxWjUOK6FYV3bDZFjc2Fn2WM4+tEloOrcNLpASNVPOh332341GjxwVAg
USYLkh1Co7yqaQ1c830CD58XxmYsR/x0B8etL1bYQtZDJYITMa649tYCYtrAvHSw
GsgwDD67yaICruIJqwPAtz5+fkoF9xnlltz0UTUSsmzrlGkdbkHSnVrvOdgbvJEz
60UM1p7idKEcR5SCkNuLD9hYJaD/C7qRhyxYfjyjPlwtSSb9aSY+TBa0t/lRmPtU
q5N/EHRdm3CmTVqE3eT1IOoRsFibYfRJNJnkqgmVZdoHm/QhTOmuyK1SeBbB3P7F
Lnw7hS6aB7RMISg8rWiPGap+QVO8lzMjIXhd5BDu1Kxkk5dr7FLOC37aSC7X52VT
QcwvYlYPSV5s+ivFk/uKtB8L3k9rHsYHHkP13lli2a9ELy+KTv0SFp0Q8ClaC9Xm
RoWXKQxrTV2wxub4/V0UfSSUfct0mHfTFDltpDPAfaEL/ARgvIUI2SD6WXvfbums
yG2fS4Rr9NHr9bgkZJqt+anNGQOmly7654ecckD+Nj2PtmXnowBOrub91VvXSfKB
TKHZ7xmX9GOnk6qsFZppiXIxDXR9zYeLb5Ks0hCF3XDhXu8DAkS4vbmPm7BIn6fu
N1niCzihe187mJ3bKAj5rLSHpbBrMt/XcbaL+eNz7xIEtpWQjk+8qFUaxl5NxIX1
KtCwLhxUeZUCKHIv/cGOJKPANfdhN2SYHasFJvJ1Jts4us4JJg4rBjH50hiYaQ3W
OhGSQP7kvcKqMzHBhSXBKPpnwBtJ+tkXy5IpeMHUrREpGt39EOO4oiiyRYSrrm2T
GswOJwvv4Z32hreR2eAgfPAXZW3R6MtGA07xN3mqPtV1IO+izegbflVpQEi8Zlq3
KfT/ljoLiDTKdIsrlfIgtgec+G5SH6znO9Kv1IidYg9wtdL6G2bsTP9pCV24bCem
srTw3NDKfTK56Yu1ESQpf0WYMTzpGjzDhyHW86F5ej+jJV2rd4kicxvy0HJXOGbI
iHPvKVVoA7bMvvfVARoMIlqiQ5gzm41+
=z4VQ
-----END PGP MESSAGE-----
 
--Apple-Mail=_4C5344B9-76FE-43EF-8620-073841EBF944--

Reference Pages and Additional Reading:

GPGTools First Steps
Secure Email in Thunderbird and Apple Mail with GPG

First, I apologize for being off-the-grid for so long.  I recently started working for a new company and am deep in the throes of the start-up life.  I started working on this incredibly long article about building the ultimate Ubuntu-based development machine and got kind of lost in the details once I hit about 5,000 words.  It’s a very long article.

I’ve also been incredibly busy in mongo-development and have learned an entire catalog of new tips and tricks that I can’t wait to share — just trying to find the time to document everything!

So, in the meantime, I thought I’d share this little useful tip.

Recently, I’ve moved my development platform over to Ubuntu 11.10 — out of necessity rather than choice — and have been learning a lot about making the transition from the Mac (where we have Linux smugness without the administrative overhead) to to a more hands-on platform.

I initially chose Ubuntu because I wanted something a little more automated that RedHat or CentOS when it came to system updates and configuration — I need concentrate of developing software, not administrating the development platform.  Ubuntu fits this requirement quite nicely.

In making the switch over from Mac to Ubuntu, most of my development tools ported straight across — PHPStorm, Chrome, Xdebug.  Other tools are native such as mongo, apache2, mysql, and PHP5.  And still other tools are facsimiles of the Mac utilities such as guake, evolution, keepassX and others.

The one tool/utility I miss the most, however, is 1Password from AgileBits software.  Seeing how they have absolutely no plans for porting their platform over to Linux, I opted to use KeepassX as my password manager program.  Not as fancy and no browser integration, but it gets the j0b done.  And, of course, I can store my key files in the cloud using Ubuntu-One.

Until I discovered 1PasswordAnywhere.

1PasswordAnywhere is a web-based version of 1Password that gives you access to your accounts via a web portal of sorts.

I say “of sorts” … while you’re loading an HTML page into your browser, you’re doing so as a file request and not as a http request.  The difference is that your “secure” data is served locally to your browser across your file system and not over the wire via the internet.

The first thing to do is locate your 1Password storage.  For most of us, this should be on dropbox.

Open your dropbox folder from the top-menu and select your 1Password folder.

When you open the 1Password folder you want to look for the file 1Password.html.

Don’t double-click on this file yet because, if you do, you’re going to see this error message shown in the image below:

This error message (and sorry if this layout is a bit confusing) requires that you disable a setting in your browser (and I’m assuming you’re using Chrome) for the html page to be displayed correctly.

By default, file:// URIs cannot read other file:// URIs.  You can disable this by launching Chrome from the command line with the option: –allow-file-access-from-files as shown below in the code block:

When you start Chrome with this switch, and you double-click on your 1Password.html file, you should see 1PasswordAnywhere displayed successfully in your browser window!

Couple of notes –

• 1PasswordAnywhere locks itself after a minute requiring your master password to unlock
• There’s still no browser extension allowing you to auto-submit your credentials to a site

What you do get, however, is access to all of your 1Password data that you’ve saved from your other devices.  For me, personally, this is substantial and well worth the effort of loading Chrome from the command line.

It’s not that I have anything against KeepassX, mind you.  It’s just that I don’t want the redundancy and synchronization issues of managing hundreds of logins and other account data.

I promise that I’ll get the uber-article out on making the ultimate development platform on Ubuntu soon – I’m waiting until the 12.04 release which should be dropping within a week or two.

Hope this helps!

I was going insane working on a buddypress install for our intranet’s tech-blog because every page I attempted to load into WordPress following the Buddypress install resulted in a 404 error.

Eventually, after several (re)installs, I tracked the issue down to a mod_rewrite issue.  Because I’m using sub-directories through a <directory> tag in my apache configuration, my .htaccess file looks different from the normal WP/BP installations:

Source code

# BEGIN WordPress
<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteBase /techblog/
    RewriteRule ^index\.php$ - [L]
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule . /techblog/index.php [L]
</IfModule>

I kept mucking about in this file thinking that there was something in my configuration that was incorrect causing me to 404 whenever I left the main page.

My boss inserted an error into the .htaccess file and, when we reloaded the page, we didn’t generate a 500 error — the site’s main page loaded, but the sub-pages were still broken.  That we injected an intentional error and the page loaded indicated that mod_rewrite was ignoring .htaccess and so the problem had to be further up in the food chain.

My <directory> alias was incorrect in two places:

Source code

Alias /techblog /var/www/techblog
 
<Directory "/var/www/techblog">
    Options Indexes MultiViews
    Order allow,deny
    Allow from all
</Directory>

First, I learned that you have to explicitly state AllowOverride All – the default setting was: AllowOverride None.  Which tells Apache to ignore the .htaccess file.  Deleting that line from my configuration did not have the anticipated effect of telling Apache to not ignore the .htaccess file; you must explicitly state AllowOverride All.

The second thing I learned was that you have to add FollowSymlinks to the Options directive so that your permalinks will be honored.  My configuration file now looks like this:

Source code

Alias /techblog /var/www/techblog
 
<Directory "/var/www/techblog">
Options Indexes MultiViews FollowSymlinks
AllowOverride All
Order allow,deny
Allow from all
</Directory>

When I restarted Apache, the BP site correctly loaded the sub-pages without error.

Today I was putzing around in the geo-spatial collection when I noticed that I had an unhappy over one of the column names within the collection.

In the mySQL world, changing a column name is pretty straight-forward courtesy of the alter table command.

Mongo…not so much…

<BEGIN_UNRELATED_SIDE_RANT>

The Mongo documentation is normally the first place most of us go when we’re looking for help in using our favorite noSQL database.

Why?

Well…because that’s usually where Google directs us to go and also because there just isn’t a whole lot of documentation out there on the subject to begin with.

The mongo (10gen) documentation is pretty good.  It’s not, however, excellent.  And I can articulate the reason why.

It’s pretty easy to identify documentation written by engineers as opposed to documentation written by everyone else (on the planet).  And not because of technical content or the (ab)use of really big and impressive-sounding jargon.

No – it’s because most engineering-authored documents are written using a solution-based voice instead of a problem-based voice.

Think about it:  when I have to go to the man-page for help, it’s because I have a problem.  If I had a solution, I would be writing a blog post.    But since I have a problem, I need the man-pages, online docs, whatever, to help me figure-out a solution.

Engineering documents are written from a solution perspective:  the document assumes you possess some bit of arcane lore (which is probably just exactly that little bit of lore that you’re missing which has caused your trip to the documentation vault) and everything that is explained within the document all hinges on this piece of knowledge which the author, albeit with the finest of intentions, assumes is already firmly in your mental possession.

And that’s why I usually don’t like 10gen’s documentation.  But, like I said earlier, it’s the only game in (Google)town.

<END_UNRELATED_SIDE_RANT>

In mongo, to change the name of a column within a collection, you first have to be on a release of mongodb 1.7.2 or later.  Since most of us bleeding-edge, early-adopter types are all 2.x versioned, this shouldn’t be an issue.

This page from 10Gen is the update page and, within, talked about the $rename modifier to the update command.  What the section doesn’t say, because it’s assuming you’re wanting to update records and not schema, is how to apply a change to all of the records in your collection.

In my case, I have a column-name which I fat-fingered the name right out it’s camel-case:  CountryID instead of countryID.  (And, yes, OCD-peeps, I know that it’s not strictly camelCase, thank-you!)  I want to spin through all 3.7 million rows in my collection and rename this column…

Source code

> db.geodata_geo.update( {} , { $rename : { 'CountryID' : 'countryID' }}, true, true );

So what we have here is the update command to the collection (geodata_geo) and four parameters:

1. {} — the empty set (this is what’s missing from the 10gen doc) implying to do whatever to each record in the collection
2. $rename — the modifier to the update command which, in this case: replace ‘CountryID’ with ‘countryID’
3. false — indicates to allow upserts if the record does not exist
4. true — multi option:  means to apply command to all records since, by default, the update() quits after updating the first record

And I run this command and mongo goes off (whirr…whirr … I have two-node replication…) and renames the column in my collection!

What it didn’t do was update my index. 

So, after my column-renaming completed, I needed to drop the index(es) that had ‘CountryID’ as members and re-index the collection to reflect the new column name.

Executing getIndexes() confirmed that my mongo world was back in it’s correct orbit and life, once again, was good.

Star Wars – The Old Republic (SWTOR) has pretty much dried-up my technical productivity since it’s release last week.  Staying true to my altoholic self, I’ve now got 6 teenage-level toons running around.

I’m surprised, after playing fantasy-based RPGs for the last decade, how much I am enjoying this game.  True – it is just another MMO similar in aspects to WoW or Rift.  But it’s that change of the menu from fantasy to sci-fi that’s making the transition so enjoyable.

I do wonder what’s happening on Telara…

Anyway, I play on a 27′ iMac i7 using Bootcamp with Windows-7 Ultimate 64-bit edition installed to play both SWOTR and Rift.  Installation was really easy – albeit time consuming — and once I got the right (as in correct) combination dialed-in for my graphics settings, game play is very smooth and responsive.

Here’s a screen shot of my settings.  Basically, I’m running in full-screen windowed-mode, custom-settings.  I have high-texture settings and high character detail, medium texture anisotropy, and low shader complexity and low shadow quality.  Grass quality is set to 100% and tree quality is set to 50%.

I’ve reduced the size of the image a bit; click on it to view it in full-screen.

In spaceport, my resting framerate (to enable fps display, press CTRL+SHIFT+f and the fps is displayed in the lower-left corner of the screen) is about 60 fps on average.

Most of the time when I’m bouncing about, I’m usually around 50fps — still very smooth with no noticeable lag, jerkiness or stuttering.

FWIW — I play on Drooga’s Pleasure Barge (server) and my main is named Mal’loathe.  Sith, of course.

Why’s my mongodb query so slow?

I got my geospatial collection set-up — I am running some really great queries making sure that the locations I am pulling aren’t in any sort of cache, and I am just blown-away by how fast data is being returned.

The problem is:  when I query the collection to pull up the requisite lon/lat data by name:  city & state, or city & country, the query seems to take seconds to complete!

I set-up the table correctly…I indexed the crap out of all my columns…a week or two ago, I was at the mongoSV 2011 in Santa Clara and learned some really cool stuff about queries, indexing, and performance management, so let’s dig-out the notes and see where I went wrong.  Because I strongly doubt that the problem is in mongo but, rather as we used to say in technical support: this is a PBCK issue…

The first thing I want to do is run an explain against my query so I can see mongo’s query plan for my query.  This should provide me with a starting point for trying to figure out what went wrong.

> db.geodata_geo.find({ cityName : “Anniston”, stateName : “Alabama” }).explain();

By adding the trailing function: .explain(), I’m requesting that mongoDB return the query-plan to me instead of executing the query.  I hit enter to launch the explain() and get back the following output:

> db.geodata_geo.find({ cityName : “Anniston”, stateName : “Alabama”}).explain();
{
“cursor” : “BasicCursor“,
“nscanned” : 3691723,
“nscannedObjects” : 3691723,
“n” : 1,
“millis” : 2269,
“nYields” : 0,
“nChunkSkips” : 0,
“isMultiKey” : false,
“indexOnly” : false,
“indexBounds” : {

}
}

The important information, I bold-faced in the query output (above).   What this output is telling me is that I’ve using a “BasicCursor” for my search cursor — which is indicates that, yes, I am doing a table-scan on the collection.  So, already I know my query is not optimal.  But, wait!  More good news…

The value for nscanned and nscannedObjects is the same: 3,691,723 — which coincidently is the same as the cardinality of the collection.  This number is the number of documents scanned to satisfy the query which, given it’s value, confirms that I am doing a full table scan.

millis tells me the number of milliseconds that the query would take:  2.269 seconds:  way too slow for my back-end methods() serving a REST API — unacceptable.

And then we get to the tell:  IndexOnly tells me that if the query could have been resolved by an (existing) covering index.  Seeing the value false here tells me that the collection has no index on the columns I am scanning against.

What?!?  I know I indexed this collection…

So, I run db.geodata_geo.getIndexes() to dump my indexes and … I … don’t see my name columns indexed.  Oh, I remembered to index the the ID and Code columns…but somehow, indexing the Name columns completely slipped past my lower brain-pan.

I add these indexes to my collection:

> db.geodata_geo.ensureIndex({ cityName : 1 });
> db.geodata_geo.ensureIndex({ stateName : 1 });

And then I rerun the query plan and see the following output:

> db.geodata_geo.find({ cityName : “Anniston”, stateName : “Alabama”}).explain();
{
“cursor” : “BtreeCursor cityName_1″,
“nscanned” : 2,
“nscannedObjects” : 2,
“n” : 1,
“millis” : 101,
“nYields” : 0,
“nChunkSkips” : 0,
“isMultiKey” : false,
“indexOnly” : false,
“indexBounds” : {
“cityName” : [
[
"Anniston",
"Anniston"
]
]
}
}

Instead of BasicCursor, I see BtreeCursor which gives me a happy.  I also see that the nscanned and nscannedObjects values are now more realistic…seriously:  2 is a LOT better than 3.6 million something, right?  Another happy for me!

I score the third happy when I see that the millis has dropped down to 101:  0.101 seconds to execute this search/query!  Not jaw-dropping, I agree — but acceptable considering that everything is running off my laptop…I know production times will be much, much lower.

In the end, I learned that a simple tool like .explain() can tell me where my attention is needed when it comes to optimization and fixing even simple, seemingly innocent queries.  Knowing what you’re looking at, and what you’re looking for, is pretty much thick-end of the baseball bat when it comes to crushing one out of the park.

I hope this helps!

Reference Link:  Explain

I’ve recently finished a mongo collection that stores all auditing data from my application — specifically, it records every database transaction, conducted in either mySQL or mongo, assigning an event-identifier to the event, and storing the data under an event ID within a single sessionManger object.

Sounds good?

Well, I like it.   This design eliminated the need to maintain meta-data in my data tables since I can pull transaction history for any record that I’ve accessed.

The problem is that, being new to mongodb, accessing what I’ve put into mongodb isn’t (yet) as intuitive as, say, my mySQL skills are.

Sub-documents within a mongo document are analogous to the results of a mySQL join.  One of the key motivators in storing this information in mongodb to begin with was that I could de-normalize the data by storing the sub-document with it’s parent instead of having to incur the expense of a search-join-fetch later.

Traditionally, any data objects defined as a one-to-many type of a relationship (1:m) were stored in multiple mySQL tables and were accessed via some sort of join mechanism.

Mongodb breaks that traditional mold by allowing you to store a sub-document (the “m” part of the 1:m relationship) within the same document in which you’re currently working.

Using my sessionManger document, I have a document that looks something like this:

My desire is to, for every database event that is recorded, enter information about that event within the sub-document that I’ve wittily named “event”.

In my PHP code, I’ve written a sequence manager for mongo that maintains a document containing sequence values for various tables.  Think of this as the functional version of mySQL’s auto-increment feature.  I decided, then, for the sessionManager events, I would use this key sequence to obtain unique values and use those as my sub-document index.  I’d then store whatever data I needed to store using the sequence value as a sub-document key, or index:

So, when I need to add another event, I just create a new sub-document under the event key, then add the data I need to store under the sub-document index key.

Worked like a champ!

And then I asked myself:  ”So, Brainiac, how would you go about extracting event -n- from your collection?”

I went through a lot of failed query attempts, bugged a lot of people, googled and saw stuff  that led me down many plush ratholes until I finally, through some serious trial-and-error, got the answer…

> db.mytable.find( { foo : bar }, { ‘event.n’ : 1 } );

where n = the number of the event I want to find.

If I want to get all of the events for a particular document (sessionManger object), then I would write something like:

> db.mytable.find( {foo : bar}, { event : 1});

If I wanted to return all of the events for all of the objects, then I would write this:

> db.mytable.find( {}, {event : 1});

What I’ve not been able to figure out, so far, is how I can use $slice to grab a range of events within a document.  Everything I try returns the full sub-set of documents back to me.  The doc tells me that $slice is used to return a subrange of array elements, which is what I thought “event.n” was but, apparently, it’s not.  (I think it’s an object (sub-document) which is why $slice fails for me.)

It’s not a big deal because, programmatically, I can grap the entire sub-document from it’s parent and parse in-memory to get the desired record.  And, if I know what the value for -n- is, then I can fetch just that one sub-document.  So, I’m ok for now.  However, please feel free to enlighten me with your expertise and experience should you see where I am failing here, ok?

I’ve been working to build, and re-build, a geospatial table for work.  There’s been a lot of challenges in this project for me as this is the first time that I’ve had to architect db designs incorporating mongodb with mySQL.

The mongo geospatial repository will be replacing several tables in the legacy mySQL system — as you may know, mongodb comes with full geospatial support so executing queries against a collection (table) built in this manner is shocking in terms of it’s response speeds — especially when you compare those speeds to the traditional mySQL algorithms for extracting geo-points based on distance ranges for lat/lon coordinates.  The tl;dr for this paragraph is:  no more hideous trigonometric mySQL queries!

What I learned in this exercise was that the key to architecting a mongo collection requires you to re-think how data is stored.  Mongo stores data as a collection of documents.  The key to successful thinking, at least in terms of mongo storage, is denormalization of your data objects.

Let’s use a standard customer object as an example.  Every customer has at least one phone number.  Most, if not all, customers have more than one phone number.  We could define several columns in the customer table for the phone numbers: workphone, homephone, cellphone, otherphone and store the data that way.  Problem is that we will eventually hit the wall where we have the need to store numbers for which we don’t have columns pre-defined:  faxphone, skypephone, tddphone, vrsphone, etc.

RDBMS design demands a normalization of this 1:M data design by requiring a second table to store just phone numbers for each customer.  The phone table would have a primary key (id), the customer id, the customer phone number and perhaps a short, descriptive, field explaining the purpose of this number.  To get the phone data for a customer, then, you’d simply query (or join) the phone table based on the customer ID to get all the phone tuples for that customer.

Mongo, on the other hand, sees every customer as a document.  Think of each customer in your collection as a piece of paper.  You want to go into your collection and retrieve on piece of paper upon which has all the customer data.  So, for example, you retrieve the document for “John Smith” and on this document, it lists several key-value pairs, underneath an array called phone:

phone : {
home : (408) 123-4567,
work : (415) 123-4567,
cell : (312) 765-4321
}

…and so on…

Mongo stores the document for this, or any user, by de-normalizing the data relationships within the customer object.  These relationships can be maintained as sub-arrays within the document.  Because mongo is schema-less, every customer object isn’t required to have all the possible combinations of phone numbers.  So, if you were to do a search where you pull-up all customers with fax numbers, our Mr. Smith would not appear in this list since he has no fax number listed in his phone array.

See?

This first step towards clarity in mongo architecture, then, is to think of all the data when you design a class object and include that data within a single document.  Data that was stored, in traditional RDBMS relation-based tables, is incorporated into the document as sub-arrays to the document.

But, you’re asking, what if you want to later add a fax number to John Smith’s phone collection?  Can you do that?

Sure!

Again, this is the inherent strength in mongodb — it’s schema-less!  Adding another number to the existing collection of phone numbers, or adding a new “column” to the document itself, requires only that you update that document.  That’s it!

So, returning back to the geospatial build, I used mySQL pull the legacy data, and collect the updated catalog tables into a single database.  Then I built new tables that (a) eliminated columns I no longer needed and, (b), de-normalized the data so that every tuple in every table reflected all of the data.

I then combined the five tables into a single table under a new primary-key value and then imported this data directly into a mongo collection.  This took several hours as my collection has over 3.6 million rows.

Once I had the collection in mongo, I made a mongo-dump of the collection so that I’d could recover back to this point in-case anything went south.  (Which it did…)

I executed a PHP script I wrote to scan the mySQL table, get the tuple by the newly-created primary key, and then create the sub-array in the mongo collection for the geospatial data.  See, in order to impose a geospatial index, your lat/lon data has to be a sub-array within the primary collection.  There’s no way I’ve yet discovered to import data over from either a flat (csv) file, or directly from mySQL, so that it creates your sub-array automagically.  Hence, the home-brew PHP script to parse through the mySQL records and build (insert) the sub-array in the newly-created mongodb collection.

(Side note:  I was careful to maintain the maximum mantissa values for the lat/lon fields by importing initially, into mySQL as varchar(255) fields — this kept my 13-digit mantissas.  When I imported the data into mongodb, mongo converted these values into doubles and kept the precision.  However, my PHP program, casting these values to either (float) or (double) converted (round) the matissa to 7-digit precision.  Suitable for task?  Yes.  Annoying to lose this data?  Yes.  If you have a solution to this, please leave me a comment at the end of this article.  Thanks!   )

The next step was to add the geo-spatial index to the collection:

> db.geodata_geo.ensureIndex( { loc : “2d” } );
point not in interval of [ -180, 180 )

What?

This error message was telling me that my data was out of range of the acceptable lat/lon values!

I tried searching for the data culprits:

> db.geodata_geo.find( { "loc" : { $exists : true }}).count();
3685667
> db.geodata_geo.find({"loc.lon" : {$lt : -180}}).count();
0
> db.geodata_geo.find({"loc.lon" : {$gt : 180}}).count();
0
> db.geodata_geo.find({"loc.lat" : {$gt : 180}}).count();
0
> db.geodata_geo.find({"loc.lat" : {$lt : -180}}).count();
0

These queries were telling me that while I have over 3.6 million records indexed, none are outside of the -180,180 boundaries.

> db.geodata_geo.find({"loc.lat" : {$gt : -180}, "loc.lon" : {$lt : 180}}).count();
3685663

> db.geodata_geo.find( { "loc" : { $exists : true }}).count();
3685667

These queries tell me that I have a delta of 4-records that exists outside of the -180, 180 boundary.

Wait...wut?

I focus on the $gt/$lt and wonder if I have an "edge" case.  Given that I've lost 6-digits of my mantissa, I wonder if mongo has rounded my data to my edge cases of 180:

> db.geodata_geo.find({"loc.lon" : 180 });

And I get back exactly four records that have a lon-value of exactly 180:

"loc" : { "lon" : 180, "lat" : -16.1499996 }

This, to me, seems to be an error in how mongodb indexes geospatial data.  If 180 is an allowed value for lat/lon, then why throw the error when you ensure the index?  I decide to handle this rounding problem by expanding the allowable limits of my query:

> db.geodata_geo.ensureIndex({ "loc" : "2d" }, { min : -180, max : 181 });
> db.geodata_geo.getIndexes();
[
	{
		"v" : 1,
		"key" : {
			"_id" : 1
		},
		"ns" : "dev_honeybadger.geodata_geo",
		"name" : "_id_"
	},
	{
		"v" : 1,
		"key" : {
			"loc" : "2d"
		},
		"ns" : "dev_honeybadger.geodata_geo",
		"name" : "loc_",
		"min" : -180,
		"max" : 181
	}
]

And I see that my geospatial index has been created.  Now, to test:

> db.geodata_geo.find( { loc : {$near : [-50,50] } } ).limit(5);

And it immediately returns five records (Elliston, Bonavista, Elliston Station, Catalina and Port Union, Division #7,  in Canada) that I asked for.

My geospatial index is complete!  Now, all I need to do is add my regular indexes for keyed searching and export the table off my development environment.

We’re updating a large dataset at work — there’s about an 18% increase in the number of tuples in the new dataset spread across a highly-normalized 8 or so tables.

I have to port the new data into a (more) efficient table structure — so I’m de-normalizing the heck out of the data reducing the schema from eight tables to a single table.

In the old architecture, four of the tables have unique key values that were imposed on the data during the original port.  So, to maintain application compatibility in the data catalog, these key values have to be maintained.  Additionally, new tuples of data have to be added to the data set and new (old) key values assigned.

In porting over one of the updated tables, which uses a string-code as the primary key, I first export the old table columns (pkey, str_code) into a temp table and then add an auto_increment int() column to the new table marking the column as default = null.  I then do a simple update-join to bring over the old pkey values based on the native str_code.

This leaves me with a new numeric column that has a variable number of NULL values (representing the delta of the new-data import) interspersed with the legacy data pkey -> str_code values.

The problem is: how do I convert the NULL pkey fields to a meaningful value that maintains the auto-increment without causing mysql to totally freak?

The first thing I do is get the max-value of the pkey:

select max(pkey_field_name) from table_name;

+---------------+
| max(pkey_...) |
+---------------+
|          4162 |
+---------------+
1 row in set (0.00 sec)

Next, I need to reset the auto_increment value of the column because, since the column is just a numeric column, it currently defaults to zero.  Attempting to convert the column on-the-fly to auto-increment will cause mysql to spit and error out about duplicate primary key values…

mysql> alter table table_name auto_increment=4163;

Query OK, 3965 rows affected (0.05 sec)
Records: 3965 Duplicates: 0 Warnings: 0

Now that I have the auto_increment reset, I can convert the column to auto_increment type and, in the process of converting, mySQL will re-number the NULL column key values using the new auto_increment value so that my numbering scheme is seamless.

When I query the data back, I can see that my new column is completely re-ordered with the legacy data maintained and the new data correctly keyed.

Resetting the auto-increment key is a handy little trick to know — I also use it when building test datasets and I need a fast way to reset my table values.

Hope this helps!

Note:  Here’s the complete and full steps to successfully complete this operation.

(DDL for admin1 table)

CREATE TABLE `admin1` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `field1` varchar(255) NOT NULL,
  `field2` varchar(255) NOT NULL,
  `field3` varchar(255) NOT NULL,
  `field4` varchar(255) NOT NULL,
  `field5` varchar(255) NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=4163 DEFAULT CHARSET=utf8

(DDL for admin1ll table)

CREATE TABLE `admin1ll` (
  `field1` int(11) NOT NULL,
  `field2` varchar(255) NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8

Step 1:  Create a new integer column in the table

ALTER TABLE `meridian`.`admin1` DROP COLUMN `id`, ADD COLUMN `id` int UNSIGNED FIRST, CHANGE COLUMN `field1` `field1` varchar(255) NOT NULL AFTER `id`, CHANGE COLUMN `field2` `field2` varchar(255) NOT NULL AFTER `field1`, CHANGE COLUMN `field3` `field3` varchar(255) NOT NULL AFTER `field2`, CHANGE COLUMN `field4` `field4` varchar(255) NOT NULL AFTER `field3`, CHANGE COLUMN `field5` `field5` varchar(255) NOT NULL AFTER `field4`;

Step 2:  Update the new column by inserting the previous-tables pkey values

update admin1, admin1ll
set admin1.id = admin1ll.field1
where admin1.field1 = admin1ll.field2

Step 3: Update the Auto_Increment value:

> select max(id) from admin1;

4030

> alter table admin1 auto_increment=4031;

Note:  the row count does not imply or set the auto_increment value!

Step 4: Reformat Column

Set column to auto_increment, unsignent, not null, primary key

Step 5:  Validate!

select *

from admin1, admin1ll

where admin1.id <> admin1ll.field1

and admin1.field1 = admin1ll.field2

> Empty set (1.70 sec)

First, let me begin by saying that I have been an Apple fan for a long time. In addition to using their computers, I was a first generation adopter of both the iPod and the iPhone.  I’ve gone through the original iPhone, the 3G and the 4.

What I like about the iPhone can be summed up by saying that it represents exemplary engineering on top of a ghastly deployment.

After all those years of being an iPhone user with AT&T, I could no longer justify paying for AT&T’s sub-standard cellular services.  The times I could complete a conversation on my iPhone without dropping the call, or losing call quality was an extremely rare event.  (And I live in the Bay area, home to AT&T which is touted as one of their major coverage areas….)

AT&T’s cellular service is simply pathetic.  As a matter of fact, I deemed it justifiable to pay the contract cancellation penalty with AT&T to come to Verizon.  I never realized how bad AT&T service was, however, until switched to Verizon.   Within my first week of using my Thunderbolt, I experienced NO dropped calls, NO poor signals, NO service interruptions of any kind.  (Unless I was connected to an AT&T cellular customer…)

And, while Apple is, in my opinion, the best hardware engineer in the business, the other factor in my decision to move phone platforms was that Apple never quite got the whole synchronization thing down. (Pre-iCloud)  I really don’t need seven different listings for my veterinarian in my address book.  Syncing services with Google have been working for me perfectly – and I don’t have to pay them $99 per year to screw up my address book contents or my calendar.

So I dropped .me and my iPhone for a Google phone living in a Google world.  Since the Android requires a Google account, (Hey! I have one of those!), setting up my phone required that I only provide it with my account information and everything from that point was auto-magically configured for me.  Insta-integration with all my Google-based services.  Plus really cool stuff like Google Voice for messaging.

….

It’s now been six months with the Android. Apple has just released their iPhone 4S…and, within a day, I find myself browsing the provider’s pages looking/comparing contracts and service offerings.  What the hell am I doing?!?

The shine on this Android HTC Thunderbolt phone is definitely gone.  While I like most of the Verizon services, specifically the quality of the cellular coverage, I am really dissatisfied with some of their processes and, as far as the phone is concerned, the HTC Thunderbolt is a complete a total piece of crap.  I will NEVER, EVER buy another phone from HTC every again.

Problems with Verizon and the HTC Thunderbolt Nobody Talks About:

1. Crapware

I rooted my Android within a week of getting it.  Verizon pre-loaded the Thunderbolt with an amazing amount of crapware that they don’t allow you to delete off their phone.  Seriously bad software.  That does nothing except eat tons of space in my memory store.  Once I rooted the phone (similar to the jail-breaking process for the iPhone), I was able to delete that bloatware and regain my lost storage for other applications.

2. Say My Name, Bitch!

Of course there’s a problem with rooting your phone — and that’s dealing with Verizon’s never-ending attempts to force software updates down your throat and to your phone.  Should you make the tragic mistake of leaving your phone “on” (which I do when sitting at my desk with the phone plugged into the charger) then Verizon assumes control of your phone by forcing your phone to accept updates over the network.

Since you’ve rooted your phone, said updates (which are image zip files stored to your cache) will not install after the download completes and the phone reboots itself.  All without any confirmation or interaction from your part.  Special, no?

When your phone reboots, you’re presented with the broken-android symbol and you have to go into your root-tools menu to delete the cached files from your phone.  This removes the forced-download and allows you to reboot your phone into it’s previously rooted state.  Of course, leave your phone on for too long and here it comes again!

There’s no “off” switch to disable the forced downloads.  Verizon’s attitude, gleaned from the forums, is that: “It’s our network.  Suffer, bitch.”

3. Random Reboots and Disappearing Apps

My co-workers claim that my phone re-booting itself (without an “upgrade” being pushed down) is because I rooted my phone.  After reading the complaints about the HTC Thunderbolt out on the etherstream – I beg to differ.  I think the HTC/Verizon mash-up operating system is just so crappy it crashes and forces a reboot.  I’ve noticed that this happens when the network flips around a lot.  I’ve also noticed it booting for no apparent reason.

What’s also special is that apps just disappear off your phone following a reboot.  Once your phone restarts, you have to give it several moments of 4G time to restore whatever apps it randomly deleted.  Totally weird behavior.  It’s almost like using a Windows operating system.

4. I’ve Lost My Network and I Can’t Get Up

Several times with this Thunderbolt I’ve noticed that I’m stuck in 1G mode.  I try toggling the mobile network connection off/on to reset it, but it always comes back to the 1G network.  This occasionally happens when it gets stuckin 3G mode as well.  (Funny, I’ve never seen 2G…)

The only way I’ve found to fix this problem is to force a restart.  When the phone regains consciousness, it happily joins the 4G network.

5. Sucks like a Starving Vampire

Granted, the Gingerbread update is supposed to fix a lot of the issues with the HTC Thunderbolt’s ability to drain your battery faster than a starving vampire in a blood bank.  I even upgraded, spending about $50, for a uber-battery, doubling the phone’s weight and thickness.  It’s worth it, though, having a battery that can last me on the train ride between San Jose and San Francisco.

There’s entire web pages devoted to tricks and tips to prolonging the battery life on this phone all of which basically involve crippling, or at least diminishing, all of the features that justified the purchase of your phone in the first place.

I’m really hoping Gingerbread offers better battery life as, since I don’t live in a winter-zone anymore, it’s a shame to waste the hand-warming features of a rapidly depleting battery.

So…I un-rooted my phone so that I could get the Gingerbread update, replacing Froyo on this phone.  I have no idea why it takes Verizon so long to roll-out these updates.  Perhaps their visual basic programmers are having a hard time with all the Android unixey stuff.  Who knows?  I mean, you have to make sure that the user can’t delete the fucking golf demo, right?

When Gingerbread was finally available for the Thunderbolt, the update lasted all of a day, if that, before Verizon yanked the update from the download stream.  It was as if they were like: QA testing?  We’ve heard of that…  The update was so bug-ridden that it was disabling or severely-impairing phone functionality.

Now, as of yesterday, they’re starting to push the Gingerbread update back out to the users.  At a time when Google is announcing the Ice Cream Sandwich update (the successor to Gingerbread), Verizon, after one false-start, is now only 1 release behind on the operating system.

Tell me — why am I paying premium rates for a phone Verizon and HTC can’t keep current?

So, as soon as I come into a little cash, I think I will call Verizon customer services (snicker) and complain to them about this phone and their inability to provide a stable (or current) operating system platform.  I’d like to negotiate them into a new Motorola Android phone…Verizon seems ti play better with Motorola — timely updates, better hardware, non thermo-radioactive battery, etc.

I want to stay in the Google universe because everything works there.  The iPhone is looking sexy — but it’s still not 4G…not yet…