Featured Posts

BuddyPress -- Pages failing to load (404) I was going insane working on a buddypress install for our intranet's tech-blog because every page I attempted to load into Wordpress following the Buddypress install...

Read more

BuddyPress -- Pages failing to load (404) I was going insane working on a buddypress install for our intranet's tech-blog because every page I attempted to load into Wordpress following the Buddypress install...

Read more

Renaming mongodb Columns Today I was putzing around in the geo-spatial collection when I noticed that I had an unhappy over one of the column names within the collection. In the mySQL world,...

Read more

SWTOR: iMac Graphic Settings for 30-70 fps Framerate Star Wars - The Old Republic (SWTOR) has pretty much dried-up my technical productivity since it's release last week.  Staying true to my altoholic self, I've now got...

Read more

Why is my mongo query so slow? Why's my mongodb query so slow? I got my geospatial collection set-up -- I am running some really great queries making sure that the locations I am pulling aren't in any...

Read more

Subscribe

BuddyPress — Pages failing to load (404)

Category : Technical
No Gravatar

I was going insane working on a buddypress install for our intranet’s tech-blog because every page I attempted to load into WordPress following the Buddypress install resulted in a 404 error.

Eventually, after several (re)installs, I tracked the issue down to a mod_rewrite issue.  Because I’m using sub-directories through a <directory> tag in my apache configuration, my .htaccess file looks different from the normal WP/BP installations:

# BEGIN WordPress
<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteBase /techblog/
    RewriteRule ^index\.php$ - [L]
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule . /techblog/index.php [L]
</IfModule>

I kept mucking about in this file thinking that there was something in my configuration that was incorrect causing me to 404 whenever I left the main page.

My boss inserted an error into the .htaccess file and, when we reloaded the page, we didn’t generate a 500 error — the site’s main page loaded, but the sub-pages were still broken.  That we injected an intentional error and the page loaded indicated that mod_rewrite was ignoring .htaccess and so the problem had to be further up in the food chain.

My <directory> alias was incorrect in two places:

Alias /techblog /var/www/techblog
 
<Directory "/var/www/techblog">
    Options Indexes MultiViews
    Order allow,deny
    Allow from all
</Directory>

First, I learned that you have to explicitly state AllowOverride All – the default setting was: AllowOverride None.  Which tells Apache to ignore the .htaccess file.  Deleting that line from my configuration did not have the anticipated effect of telling Apache to not ignore the .htaccess file; you must explicitly state AllowOverride All.

The second thing I learned was that you have to add FollowSymlinks to the Options directive so that your permalinks will be honored.  My configuration file now looks like this:

Alias /techblog /var/www/techblog
 
<Directory "/var/www/techblog">
Options Indexes MultiViews FollowSymlinks
AllowOverride All
Order allow,deny
Allow from all
</Directory>

When I restarted Apache, the BP site correctly loaded the sub-pages without error.

 

Renaming mongodb Columns

Category : Technical
No Gravatar

Today I was putzing around in the geo-spatial collection when I noticed that I had an unhappy over one of the column names within the collection.

In the mySQL world, changing a column name is pretty straight-forward courtesy of the alter table command.

Mongo…not so much…

<BEGIN_UNRELATED_SIDE_RANT>

The Mongo documentation is normally the first place most of us go when we’re looking for help in using our favorite noSQL database.

Why?

Well…because that’s usually where Google directs us to go and also because there just isn’t a whole lot of documentation out there on the subject to begin with.

The mongo (10gen) documentation is pretty good.  It’s not, however, excellent.  And I can articulate the reason why.

It’s pretty easy to identify documentation written by engineers as opposed to documentation written by everyone else (on the planet).  And not because of technical content or the (ab)use of really big and impressive-sounding jargon.

No – it’s because most engineering-authored documents are written using a solution-based voice instead of a problem-based voice.

Think about it:  when I have to go to the man-page for help, it’s because I have a problem.  If I had a solution, I would be writing a blog post.    But since I have a problem, I need the man-pages, online docs, whatever, to help me figure-out a solution.

Engineering documents are written from a solution perspective:  the document assumes you possess some bit of arcane lore (which is probably just exactly that little bit of lore that you’re missing which has caused your trip to the documentation vault) and everything that is explained within the document all hinges on this piece of knowledge which the author, albeit with the finest of intentions, assumes is already firmly in your mental possession.

And that’s why I usually don’t like 10gen’s documentation.  But, like I said earlier, it’s the only game in (Google)town.

<END_UNRELATED_SIDE_RANT>

In mongo, to change the name of a column within a collection, you first have to be on a release of mongodb 1.7.2 or later.  Since most of us bleeding-edge, early-adopter types are all 2.x versioned, this shouldn’t be an issue.

This page from 10Gen is the update page and, within, talked about the $rename modifier to the update command.  What the section doesn’t say, because it’s assuming you’re wanting to update records and not schema, is how to apply a change to all of the records in your collection.

In my case, I have a column-name which I fat-fingered the name right out it’s camel-case:  CountryID instead of countryID.  (And, yes, OCD-peeps, I know that it’s not strictly camelCase, thank-you!)  I want to spin through all 3.7 million rows in my collection and rename this column…

> db.geodata_geo.update( {} , { $rename : { 'CountryID' : 'countryID' }}, true, true );

So what we have here is the update command to the collection (geodata_geo) and four parameters:

  1. {} — the empty set (this is what’s missing from the 10gen doc) implying to do whatever to each record in the collection
  2. $rename — the modifier to the update command which, in this case: replace ‘CountryID’ with ‘countryID’
  3. false — indicates to allow upserts if the record does not exist
  4. true — multi option:  means to apply command to all records since, by default, the update() quits after updating the first record

And I run this command and mongo goes off (whirr…whirr … I have two-node replication…) and renames the column in my collection!

What it didn’t do was update my index. 

So, after my column-renaming completed, I needed to drop the index(es) that had ‘CountryID’ as members and re-index the collection to reflect the new column name.

Executing getIndexes() confirmed that my mongo world was back in it’s correct orbit and life, once again, was good.

SWTOR: iMac Graphic Settings for 30-70 fps Framerate

Category : Gaming, Technical
No Gravatar

Star Wars – The Old Republic (SWTOR) has pretty much dried-up my technical productivity since it’s release last week.  Staying true to my altoholic self, I’ve now got 6 teenage-level toons running around.

I’m surprised, after playing fantasy-based RPGs for the last decade, how much I am enjoying this game.  True – it is just another MMO similar in aspects to WoW or Rift.  But it’s that change of the menu from fantasy to sci-fi that’s making the transition so enjoyable.

I do wonder what’s happening on Telara…

Anyway, I play on a 27′ iMac i7 using Bootcamp with Windows-7 Ultimate 64-bit edition installed to play both SWOTR and Rift.  Installation was really easy – albeit time consuming — and once I got the right (as in correct) combination dialed-in for my graphics settings, game play is very smooth and responsive.

Here’s a screen shot of my settings.  Basically, I’m running in full-screen windowed-mode, custom-settings.  I have high-texture settings and high character detail, medium texture anisotropy, and low shader complexity and low shadow quality.  Grass quality is set to 100% and tree quality is set to 50%.

I’ve reduced the size of the image a bit; click on it to view it in full-screen.

In spaceport, my resting framerate (to enable fps display, press CTRL+SHIFT+f and the fps is displayed in the lower-left corner of the screen) is about 60 fps on average.

Most of the time when I’m bouncing about, I’m usually around 50fps — still very smooth with no noticeable lag, jerkiness or stuttering.

FWIW — I play on Drooga’s Pleasure Barge (server) and my main is named Mal’loathe.  Sith, of course.

Why is my mongo query so slow?

Category : Technical
No Gravatar

Why’s my mongodb query so slow?

I got my geospatial collection set-up — I am running some really great queries making sure that the locations I am pulling aren’t in any sort of cache, and I am just blown-away by how fast data is being returned.

The problem is:  when I query the collection to pull up the requisite lon/lat data by name:  city & state, or city & country, the query seems to take seconds to complete!

I set-up the table correctly…I indexed the crap out of all my columns…a week or two ago, I was at the mongoSV 2011 in Santa Clara and learned some really cool stuff about queries, indexing, and performance management, so let’s dig-out the notes and see where I went wrong.  Because I strongly doubt that the problem is in mongo but, rather as we used to say in technical support: this is a PBCK issue…

The first thing I want to do is run an explain against my query so I can see mongo’s query plan for my query.  This should provide me with a starting point for trying to figure out what went wrong.

> db.geodata_geo.find({ cityName : “Anniston”, stateName : “Alabama” }).explain();

By adding the trailing function: .explain(), I’m requesting that mongoDB return the query-plan to me instead of executing the query.  I hit enter to launch the explain() and get back the following output:

> db.geodata_geo.find({ cityName : “Anniston”, stateName : “Alabama”}).explain();
{
“cursor” : “BasicCursor“,
“nscanned” : 3691723,
“nscannedObjects” : 3691723,
“n” : 1,
“millis” : 2269,
“nYields” : 0,
“nChunkSkips” : 0,
“isMultiKey” : false,
indexOnly” : false,
“indexBounds” : {

}
}

The important information, I bold-faced in the query output (above).   What this output is telling me is that I’ve using a “BasicCursor” for my search cursor — which is indicates that, yes, I am doing a table-scan on the collection.  So, already I know my query is not optimal.  But, wait!  More good news…

The value for nscanned and nscannedObjects is the same: 3,691,723 — which coincidently is the same as the cardinality of the collection.  This number is the number of documents scanned to satisfy the query which, given it’s value, confirms that I am doing a full table scan.

millis tells me the number of milliseconds that the query would take:  2.269 seconds:  way too slow for my back-end methods() serving a REST API — unacceptable.

And then we get to the tell:  IndexOnly tells me that if the query could have been resolved by an (existing) covering index.  Seeing the value false here tells me that the collection has no index on the columns I am scanning against.

What?!?  I know I indexed this collection…

So, I run db.geodata_geo.getIndexes() to dump my indexes and … I … don’t see my name columns indexed.  Oh, I remembered to index the the ID and Code columns…but somehow, indexing the Name columns completely slipped past my lower brain-pan.

I add these indexes to my collection:

> db.geodata_geo.ensureIndex({ cityName : 1 });
> db.geodata_geo.ensureIndex({ stateName : 1 });

And then I rerun the query plan and see the following output:

> db.geodata_geo.find({ cityName : “Anniston”, stateName : “Alabama”}).explain();
{
“cursor” : “BtreeCursor cityName_1″,
“nscanned” : 2,
“nscannedObjects” : 2,
“n” : 1,
“millis” : 101,
“nYields” : 0,
“nChunkSkips” : 0,
“isMultiKey” : false,
“indexOnly” : false,
“indexBounds” : {
“cityName” : [
[
"Anniston",
"Anniston"
]
]
}
}

Instead of BasicCursor, I see BtreeCursor which gives me a happy.  I also see that the nscanned and nscannedObjects values are now more realistic…seriously:  2 is a LOT better than 3.6 million something, right?  Another happy for me!

I score the third happy when I see that the millis has dropped down to 101:  0.101 seconds to execute this search/query!  Not jaw-dropping, I agree — but acceptable considering that everything is running off my laptop…I know production times will be much, much lower.

 

In the end, I learned that a simple tool like .explain() can tell me where my attention is needed when it comes to optimization and fixing even simple, seemingly innocent queries.  Knowing what you’re looking at, and what you’re looking for, is pretty much thick-end of the baseball bat when it comes to crushing one out of the park.

I hope this helps!

 

Reference Link:  Explain

Searching MongoDB Sub-Documents…

Category : Technical
No Gravatar

I’ve recently finished a mongo collection that stores all auditing data from my application — specifically, it records every database transaction, conducted in either mySQL or mongo, assigning an event-identifier to the event, and storing the data under an event ID within a single sessionManger object.

Sounds good?

Well, I like it.   This design eliminated the need to maintain meta-data in my data tables since I can pull transaction history for any record that I’ve accessed.

The problem is that, being new to mongodb, accessing what I’ve put into mongodb isn’t (yet) as intuitive as, say, my mySQL skills are.

Sub-documents within a mongo document are analogous to the results of a mySQL join.  One of the key motivators in storing this information in mongodb to begin with was that I could de-normalize the data by storing the sub-document with it’s parent instead of having to incur the expense of a search-join-fetch later.

Traditionally, any data objects defined as a one-to-many type of a relationship (1:m) were stored in multiple mySQL tables and were accessed via some sort of join mechanism.

Mongodb breaks that traditional mold by allowing you to store a sub-document (the “m” part of the 1:m relationship) within the same document in which you’re currently working.

Using my sessionManger document, I have a document that looks something like this:

{
_id : somevalue,
foo : bar,
event : {},
argle : bargle,
}

My desire is to, for every database event that is recorded, enter information about that event within the sub-document that I’ve wittily named “event”.

In my PHP code, I’ve written a sequence manager for mongo that maintains a document containing sequence values for various tables.  Think of this as the functional version of mySQL’s auto-increment feature.  I decided, then, for the sessionManager events, I would use this key sequence to obtain unique values and use those as my sub-document index.  I’d then store whatever data I needed to store using the sequence value as a sub-document key, or index:

{
_id : somevalue,
foo: bar,
event : {
n : {
created : dateval,
table : tableName,
schema : dbSchema,
query : lastQuery
}
}
argle : bargle
}

So, when I need to add another event, I just create a new sub-document under the event key, then add the data I need to store under the sub-document index key.

Worked like a champ!

And then I asked myself:  ”So, Brainiac, how would you go about extracting event -n- from your collection?”

I went through a lot of failed query attempts, bugged a lot of people, googled and saw stuff  that led me down many plush ratholes until I finally, through some serious trial-and-error, got the answer…

> db.mytable.find( { foo : bar }, { ‘event.n’ : 1 } );

where n = the number of the event I want to find.

If I want to get all of the events for a particular document (sessionManger object), then I would write something like:

> db.mytable.find( {foo : bar}, { event : 1});

If I wanted to return all of the events for all of the objects, then I would write this:

> db.mytable.find( {}, {event : 1});

What I’ve not been able to figure out, so far, is how I can use $slice to grab a range of events within a document.  Everything I try returns the full sub-set of documents back to me.  The doc tells me that $slice is used to return a subrange of array elements, which is what I thought “event.n” was but, apparently, it’s not.  (I think it’s an object (sub-document) which is why $slice fails for me.)

It’s not a big deal because, programmatically, I can grap the entire sub-document from it’s parent and parse in-memory to get the desired record.  And, if I know what the value for -n- is, then I can fetch just that one sub-document.  So, I’m ok for now.  However, please feel free to enlighten me with your expertise and experience should you see where I am failing here, ok?

 

mongodb, geospatial indexing, and advanced queries….

Category : Technical
No Gravatar

I’ve been working to build, and re-build, a geospatial table for work.  There’s been a lot of challenges in this project for me as this is the first time that I’ve had to architect db designs incorporating mongodb with mySQL.

The mongo geospatial repository will be replacing several tables in the legacy mySQL system — as you may know, mongodb comes with full geospatial support so executing queries against a collection (table) built in this manner is shocking in terms of it’s response speeds — especially when you compare those speeds to the traditional mySQL algorithms for extracting geo-points based on distance ranges for lat/lon coordinates.  The tl;dr for this paragraph is:  no more hideous trigonometric mySQL queries!

What I learned in this exercise was that the key to architecting a mongo collection requires you to re-think how data is stored.  Mongo stores data as a collection of documents.  The key to successful thinking, at least in terms of mongo storage, is denormalization of your data objects.

Let’s use a standard customer object as an example.  Every customer has at least one phone number.  Most, if not all, customers have more than one phone number.  We could define several columns in the customer table for the phone numbers: workphone, homephone, cellphone, otherphone and store the data that way.  Problem is that we will eventually hit the wall where we have the need to store numbers for which we don’t have columns pre-defined:  faxphone, skypephone, tddphone, vrsphone, etc.

RDBMS design demands a normalization of this 1:M data design by requiring a second table to store just phone numbers for each customer.  The phone table would have a primary key (id), the customer id, the customer phone number and perhaps a short, descriptive, field explaining the purpose of this number.  To get the phone data for a customer, then, you’d simply query (or join) the phone table based on the customer ID to get all the phone tuples for that customer.

Mongo, on the other hand, sees every customer as a document.  Think of each customer in your collection as a piece of paper.  You want to go into your collection and retrieve on piece of paper upon which has all the customer data.  So, for example, you retrieve the document for “John Smith” and on this document, it lists several key-value pairs, underneath an array called phone:

phone : {
home : (408) 123-4567,
work : (415) 123-4567,
cell : (312) 765-4321
}

…and so on…

Mongo stores the document for this, or any user, by de-normalizing the data relationships within the customer object.  These relationships can be maintained as sub-arrays within the document.  Because mongo is schema-less, every customer object isn’t required to have all the possible combinations of phone numbers.  So, if you were to do a search where you pull-up all customers with fax numbers, our Mr. Smith would not appear in this list since he has no fax number listed in his phone array.

See?

This first step towards clarity in mongo architecture, then, is to think of all the data when you design a class object and include that data within a single document.  Data that was stored, in traditional RDBMS relation-based tables, is incorporated into the document as sub-arrays to the document.

But, you’re asking, what if you want to later add a fax number to John Smith’s phone collection?  Can you do that?

Sure!

Again, this is the inherent strength in mongodb — it’s schema-less!  Adding another number to the existing collection of phone numbers, or adding a new “column” to the document itself, requires only that you update that document.  That’s it!

So, returning back to the geospatial build, I used mySQL pull the legacy data, and collect the updated catalog tables into a single database.  Then I built new tables that (a) eliminated columns I no longer needed and, (b), de-normalized the data so that every tuple in every table reflected all of the data.

I then combined the five tables into a single table under a new primary-key value and then imported this data directly into a mongo collection.  This took several hours as my collection has over 3.6 million rows.

Once I had the collection in mongo, I made a mongo-dump of the collection so that I’d could recover back to this point in-case anything went south.  (Which it did…)

I executed a PHP script I wrote to scan the mySQL table, get the tuple by the newly-created primary key, and then create the sub-array in the mongo collection for the geospatial data.  See, in order to impose a geospatial index, your lat/lon data has to be a sub-array within the primary collection.  There’s no way I’ve yet discovered to import data over from either a flat (csv) file, or directly from mySQL, so that it creates your sub-array automagically.  Hence, the home-brew PHP script to parse through the mySQL records and build (insert) the sub-array in the newly-created mongodb collection.

(Side note:  I was careful to maintain the maximum mantissa values for the lat/lon fields by importing initially, into mySQL as varchar(255) fields — this kept my 13-digit mantissas.  When I imported the data into mongodb, mongo converted these values into doubles and kept the precision.  However, my PHP program, casting these values to either (float) or (double) converted (round) the matissa to 7-digit precision.  Suitable for task?  Yes.  Annoying to lose this data?  Yes.  If you have a solution to this, please leave me a comment at the end of this article.  Thanks!   :-P )

The next step was to add the geo-spatial index to the collection:

> db.geodata_geo.ensureIndex( { loc : “2d” } );
point not in interval of [ -180, 180 )

What?

This error message was telling me that my data was out of range of the acceptable lat/lon values!

I tried searching for the data culprits:

> db.geodata_geo.find( { "loc" : { $exists : true }}).count();
3685667
> db.geodata_geo.find({"loc.lon" : {$lt : -180}}).count();
0
> db.geodata_geo.find({"loc.lon" : {$gt : 180}}).count();
0
> db.geodata_geo.find({"loc.lat" : {$gt : 180}}).count();
0
> db.geodata_geo.find({"loc.lat" : {$lt : -180}}).count();
0

These queries were telling me that while I have over 3.6 million records indexed, none are outside of the -180,180 boundaries.

> db.geodata_geo.find({"loc.lat" : {$gt : -180}, "loc.lon" : {$lt : 180}}).count();
3685663
> db.geodata_geo.find( { "loc" : { $exists : true }}).count();
3685667

These queries tell me that I have a delta of 4-records that exists outside of the -180, 180 boundary.

Wait...wut?

I focus on the $gt/$lt and wonder if I have an "edge" case.  Given that I've lost 6-digits of my mantissa, I wonder if mongo has rounded my data to my edge cases of 180:

> db.geodata_geo.find({"loc.lon" : 180 });

And I get back exactly four records that have a lon-value of exactly 180:

"loc" : { "lon" : 180, "lat" : -16.1499996 }

This, to me, seems to be an error in how mongodb indexes geospatial data.  If 180 is an allowed value for lat/lon, then why throw the error when you ensure the index?  I decide to handle this rounding problem by expanding the allowable limits of my query:

> db.geodata_geo.ensureIndex({ "loc" : "2d" }, { min : -180, max : 181 });
> db.geodata_geo.getIndexes();
[
	{
		"v" : 1,
		"key" : {
			"_id" : 1
		},
		"ns" : "dev_honeybadger.geodata_geo",
		"name" : "_id_"
	},
	{
		"v" : 1,
		"key" : {
			"loc" : "2d"
		},
		"ns" : "dev_honeybadger.geodata_geo",
		"name" : "loc_",
		"min" : -180,
		"max" : 181
	}
]

And I see that my geospatial index has been created.  Now, to test:

> db.geodata_geo.find( { loc : {$near : [-50,50] } } ).limit(5);

And it immediately returns five records (Elliston, Bonavista, Elliston Station, Catalina and Port Union, Division #7,  in Canada) that I asked for.

My geospatial index is complete!  Now, all I need to do is add my regular indexes for keyed searching and export the table off my development environment.

 

 

Converting a mySQL Column to AutoIncrement

Category : Technical
No Gravatar

We’re updating a large dataset at work — there’s about an 18% increase in the number of tuples in the new dataset spread across a highly-normalized 8 or so tables.

I have to port the new data into a (more) efficient table structure — so I’m de-normalizing the heck out of the data reducing the schema from eight tables to a single table.

In the old architecture, four of the tables have unique key values that were imposed on the data during the original port.  So, to maintain application compatibility in the data catalog, these key values have to be maintained.  Additionally, new tuples of data have to be added to the data set and new (old) key values assigned.

In porting over one of the updated tables, which uses a string-code as the primary key, I first export the old table columns (pkey, str_code) into a temp table and then add an auto_increment int() column to the new table marking the column as default = null.  I then do a simple update-join to bring over the old pkey values based on the native str_code.

This leaves me with a new numeric column that has a variable number of NULL values (representing the delta of the new-data import) interspersed with the legacy data pkey -> str_code values.

The problem is: how do I convert the NULL pkey fields to a meaningful value that maintains the auto-increment without causing mysql to totally freak?

The first thing I do is get the max-value of the pkey:

select max(pkey_field_name) from table_name;
+---------------+
| max(pkey_...) |
+---------------+
|          4162 |
+---------------+
1 row in set (0.00 sec)

Next, I need to reset the auto_increment value of the column because, since the column is just a numeric column, it currently defaults to zero.  Attempting to convert the column on-the-fly to auto-increment will cause mysql to spit and error out about duplicate primary key values…

mysql> alter table table_name auto_increment=4163;
Query OK, 3965 rows affected (0.05 sec)
Records: 3965 Duplicates: 0 Warnings: 0

Now that I have the auto_increment reset, I can convert the column to auto_increment type and, in the process of converting, mySQL will re-number the NULL column key values using the new auto_increment value so that my numbering scheme is seamless.

When I query the data back, I can see that my new column is completely re-ordered with the legacy data maintained and the new data correctly keyed.

Resetting the auto-increment key is a handy little trick to know — I also use it when building test datasets and I need a fast way to reset my table values.

Hope this helps!

 

 

Note:  Here’s the complete and full steps to successfully complete this operation.

(DDL for admin1 table)

CREATE TABLE `admin1` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `field1` varchar(255) NOT NULL,
  `field2` varchar(255) NOT NULL,
  `field3` varchar(255) NOT NULL,
  `field4` varchar(255) NOT NULL,
  `field5` varchar(255) NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=4163 DEFAULT CHARSET=utf8

(DDL for admin1ll table)

CREATE TABLE `admin1ll` (
  `field1` int(11) NOT NULL,
  `field2` varchar(255) NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8

 

Step 1:  Create a new integer column in the table

ALTER TABLE `meridian`.`admin1` DROP COLUMN `id`, ADD COLUMN `id` int UNSIGNED FIRST, CHANGE COLUMN `field1` `field1` varchar(255) NOT NULL AFTER `id`, CHANGE COLUMN `field2` `field2` varchar(255) NOT NULL AFTER `field1`, CHANGE COLUMN `field3` `field3` varchar(255) NOT NULL AFTER `field2`, CHANGE COLUMN `field4` `field4` varchar(255) NOT NULL AFTER `field3`, CHANGE COLUMN `field5` `field5` varchar(255) NOT NULL AFTER `field4`;

Step 2:  Update the new column by inserting the previous-tables pkey values

update admin1, admin1ll
set admin1.id = admin1ll.field1
where admin1.field1 = admin1ll.field2

Step 3: Update the Auto_Increment value:

> select max(id) from admin1;

4030

> alter table admin1 auto_increment=4031;

Note:  the row count does not imply or set the auto_increment value!

Step 4: Reformat Column

Set column to auto_increment, unsignent, not null, primary key

Step 5:  Validate!

select *

from admin1, admin1ll

where admin1.id <> admin1ll.field1

and admin1.field1 = admin1ll.field2

> Empty set (1.70 sec)

HTC Thunderbolt – The Honeymoon? Definitely Over.

Category : Rant, Technical, WTF
No Gravatar

First, let me begin by saying that I have been an Apple fan for a long time. In addition to using their computers, I was a first generation adopter of both the iPod and the iPhone.  I’ve gone through the original iPhone, the 3G and the 4.

What I like about the iPhone can be summed up by saying that it represents exemplary engineering on top of a ghastly deployment.

After all those years of being an iPhone user with AT&T, I could no longer justify paying for AT&T’s sub-standard cellular services.  The times I could complete a conversation on my iPhone without dropping the call, or losing call quality was an extremely rare event.  (And I live in the Bay area, home to AT&T which is touted as one of their major coverage areas….)

AT&T’s cellular service is simply pathetic.  As a matter of fact, I deemed it justifiable to pay the contract cancellation penalty with AT&T to come to Verizon.  I never realized how bad AT&T service was, however, until switched to Verizon.   Within my first week of using my Thunderbolt, I experienced NO dropped calls, NO poor signals, NO service interruptions of any kind.  (Unless I was connected to an AT&T cellular customer…)

And, while Apple is, in my opinion, the best hardware engineer in the business, the other factor in my decision to move phone platforms was that Apple never quite got the whole synchronization thing down. (Pre-iCloud)  I really don’t need seven different listings for my veterinarian in my address book.  Syncing services with Google have been working for me perfectly – and I don’t have to pay them $99 per year to screw up my address book contents or my calendar.

So I dropped .me and my iPhone for a Google phone living in a Google world.  Since the Android requires a Google account, (Hey! I have one of those!), setting up my phone required that I only provide it with my account information and everything from that point was auto-magically configured for me.  Insta-integration with all my Google-based services.  Plus really cool stuff like Google Voice for messaging.

….

It’s now been six months with the Android. Apple has just released their iPhone 4S…and, within a day, I find myself browsing the provider’s pages looking/comparing contracts and service offerings.  What the hell am I doing?!?

The shine on this Android HTC Thunderbolt phone is definitely gone.  While I like most of the Verizon services, specifically the quality of the cellular coverage, I am really dissatisfied with some of their processes and, as far as the phone is concerned, the HTC Thunderbolt is a complete a total piece of crap.  I will NEVER, EVER buy another phone from HTC every again.

Problems with Verizon and the HTC Thunderbolt Nobody Talks About:

1. Crapware

I rooted my Android within a week of getting it.  Verizon pre-loaded the Thunderbolt with an amazing amount of crapware that they don’t allow you to delete off their phone.  Seriously bad software.  That does nothing except eat tons of space in my memory store.  Once I rooted the phone (similar to the jail-breaking process for the iPhone), I was able to delete that bloatware and regain my lost storage for other applications.

2. Say My Name, Bitch!

Of course there’s a problem with rooting your phone — and that’s dealing with Verizon’s never-ending attempts to force software updates down your throat and to your phone.  Should you make the tragic mistake of leaving your phone “on” (which I do when sitting at my desk with the phone plugged into the charger) then Verizon assumes control of your phone by forcing your phone to accept updates over the network.

Since you’ve rooted your phone, said updates (which are image zip files stored to your cache) will not install after the download completes and the phone reboots itself.  All without any confirmation or interaction from your part.  Special, no?

When your phone reboots, you’re presented with the broken-android symbol and you have to go into your root-tools menu to delete the cached files from your phone.  This removes the forced-download and allows you to reboot your phone into it’s previously rooted state.  Of course, leave your phone on for too long and here it comes again!

There’s no “off” switch to disable the forced downloads.  Verizon’s attitude, gleaned from the forums, is that: “It’s our network.  Suffer, bitch.”

3. Random Reboots and Disappearing Apps

My co-workers claim that my phone re-booting itself (without an “upgrade” being pushed down) is because I rooted my phone.  After reading the complaints about the HTC Thunderbolt out on the etherstream – I beg to differ.  I think the HTC/Verizon mash-up operating system is just so crappy it crashes and forces a reboot.  I’ve noticed that this happens when the network flips around a lot.  I’ve also noticed it booting for no apparent reason.

What’s also special is that apps just disappear off your phone following a reboot.  Once your phone restarts, you have to give it several moments of 4G time to restore whatever apps it randomly deleted.  Totally weird behavior.  It’s almost like using a Windows operating system.

4. I’ve Lost My Network and I Can’t Get Up

Several times with this Thunderbolt I’ve noticed that I’m stuck in 1G mode.  I try toggling the mobile network connection off/on to reset it, but it always comes back to the 1G network.  This occasionally happens when it gets stuckin 3G mode as well.  (Funny, I’ve never seen 2G…)

The only way I’ve found to fix this problem is to force a restart.  When the phone regains consciousness, it happily joins the 4G network.

5. Sucks like a Starving Vampire

Granted, the Gingerbread update is supposed to fix a lot of the issues with the HTC Thunderbolt’s ability to drain your battery faster than a starving vampire in a blood bank.  I even upgraded, spending about $50, for a uber-battery, doubling the phone’s weight and thickness.  It’s worth it, though, having a battery that can last me on the train ride between San Jose and San Francisco.

There’s entire web pages devoted to tricks and tips to prolonging the battery life on this phone all of which basically involve crippling, or at least diminishing, all of the features that justified the purchase of your phone in the first place.

I’m really hoping Gingerbread offers better battery life as, since I don’t live in a winter-zone anymore, it’s a shame to waste the hand-warming features of a rapidly depleting battery.

 

So…I un-rooted my phone so that I could get the Gingerbread update, replacing Froyo on this phone.  I have no idea why it takes Verizon so long to roll-out these updates.  Perhaps their visual basic programmers are having a hard time with all the Android unixey stuff.  Who knows?  I mean, you have to make sure that the user can’t delete the fucking golf demo, right?

When Gingerbread was finally available for the Thunderbolt, the update lasted all of a day, if that, before Verizon yanked the update from the download stream.  It was as if they were like: QA testing?  We’ve heard of that…  The update was so bug-ridden that it was disabling or severely-impairing phone functionality.

Now, as of yesterday, they’re starting to push the Gingerbread update back out to the users.  At a time when Google is announcing the Ice Cream Sandwich update (the successor to Gingerbread), Verizon, after one false-start, is now only 1 release behind on the operating system.

Tell me — why am I paying premium rates for a phone Verizon and HTC can’t keep current?

So, as soon as I come into a little cash, I think I will call Verizon customer services (snicker) and complain to them about this phone and their inability to provide a stable (or current) operating system platform.  I’d like to negotiate them into a new Motorola Android phone…Verizon seems ti play better with Motorola — timely updates, better hardware, non thermo-radioactive battery, etc.

I want to stay in the Google universe because everything works there.  The iPhone is looking sexy — but it’s still not 4G…not yet…

mongodb.findOne() — calling with PHP variables (not literals)

Category : Technical
No Gravatar

So I’ve been doing a lot of work, for work, in MongoDB lately and I’ve learned an awful lot.  Or, depending on your point of view, a lot that’s just awful.

See, there’s not what you could even charitably call a lot of MongoDB documentation to begin with.   If you filter what is available on, oh, say, PHP implementation, well the results just dwindle to something roughly the same size as a tax-collector’s heart.

Here’s the scenario — I’ve been working on adding a mongo abstraction class on top of my base-data abstraction class — whereas said classes are extended by the table-level class instantiation.  This allows me to keep all of my query logic in the middle tier of the class design, generic and administrative functions in the base class, and table-specific stuff in the table class.  So far, so good, right?

Well, I get the mongo constructor running and, like it’s mySQL counterpart, I have an rule in every table constructor that states “if I pass a indexed field and it’s value to the constructor, then instantiate the class pre-populated with that record.”

And that’s where things start to head south.

In my constructor logic, I’m only allowing single-value key->value pairs as constructor parameters with the design intention of getting a record from the db using the pkey of the table/collection.  In other words, you get one column and one column value.  So, if you’re going to instantiate a new user object, you’d probably want to pass-in the primary-key field of a user and that field’s value:

$objUser = new UserProfile(‘email’, ‘mshallop@gmail.com’);   // instantiate a new user object with this email address

Still pretty easy.  I bang out the mySQL equivalent in nothing flat.  I hit a huge pothole when I get to the mongo side.

The method is defined as a protected abstract method in the base class – so this method has to appear in both child classes as defined in the parent:

protected abstract function loadClassById($_key, $_value);

So I have my methods defined in both the mySQL and mongoDB middle layer.  My strategy for the mongo fetch-and-return is pretty simple — once the class has been instantiated, do the following:

  1. make sure the $_key value exists in the allowed field list
  2. make sure the $_value has a value
  3. query mongodb using .findOne()
  4. store the return key->value pairs in the member array
  5. return status

That’s pretty much it.  But I run into huge problems when I get to step 3 — use the mongoDB findOne command.

The findOne method takes an array input of the key->value pair.  From the mongo command line, you’d execute something like this:

> db.session_ses.findOne({'idpro_ses' : 1})
{
 "_id" : ObjectId("4ea1af93ddc69802376b56d1"),
 "id_ses" : 1,
 "idpro_ses" : 1
}

( Just to show you that the data exists in the mongo collection…)

But, the PHP-ized version of the method is a wee bit different:

$this->collection->findOne(array(‘idpro_ses’ => 1));

All of the examples that I’ve been able to locate show using the method by invoking it using literals.  My problem is that I have the two input parameters sent to the method ($_key and $_value) and I’ve got to find a way to get the PHP version of the method call to work using variables instead of constants.  This is what didn’t work:

$this->collection->findOne(array($_key => $_value));

$this->collection->findOne(array(“‘” . $_key . “‘” => $_value));

 

$this->collection->findOne(array(“{$_key}” => $_value));

$aryData = array();
$aryData[$_key] = $_value;
$this->collection->findOne($aryData);
or
$this->collection->findOne(var_dump($aryData)); 

I thought this worked but I was wrong:

$this->collection->findOne(array(array_keys($aryData) => array_values($aryData)));

This format returned a mongo record — the problem was that it returned the first mongo record independently of any key-search criteria.

What finally worked for me was this:

            $qs = array(); // QueryStructure
            switch($this->fieldTypes[$_k]) {
                case 'int' :
                    $_v = intval($_v);
                    break;
                case 'str' :
                    $_v = strval($_v);
                    break;
                case 'float' :
                    $_v = floatval($_v);
                    break;
            }
            $qs[$_k] = $_v;
            $aryData = $this->collection->findOne($qs);

[Update]

I encountered a similar problem when trying to update records in a mongo collection — while I could update the record from the mongo command line, I did not experience the same success in trying to execute the command from within my PHP program…

$foo = $collection->find(array(‘id_geo’ => $row['id_geo']));

Consistently failed.  No exceptions were caught, and mongo’s findLastError() reported no errors in the transaction.

After several iterations of debugging and attempting various work-arounds, I stumbled upon the solution as being one of casting.  While the variable was being evaluated in the PHP array as type int, somehow this wasn’t being interpreted that way by Mongo.  Casting the variable to an integer:

$foo = $collection->find(array(‘id_geo’ => intval($row['id_geo']))); 

 generated a successful query for both the find() and my update() functions.

As I gain experience with Mongo, I expect to discover more of these little mannerisms…

Web Services with PHP & nuSoap – Part 1.1

Category : Technical
No Gravatar

Introduction

[EDIT] – This is a re-hash of a document I wrote a couple years ago.  There’s been changes to the nuSOAP library and I wanted to document the updates relative to the tutorial series.  Also, I need to fix a lot of the broken-links and code listings since I’ve changes hosting providers since this article was first written.

This article is specific to nuSOAP release 0.9.5 on 2011-01-13.  This tutorial was updated on September 22, 2011.

Once upon a time, I was tasked with developing a web-services by my boss as the integration point between our production application and Sales Force.  At this point, although I’d heard of web services … kind of … didn’t Amazon use web services for something?  Still, I’d never coded a web-services based application before.

At this point, I have to assume you are unfamiliar with the concept of web services and why you may have to create and provide a web-services offering to your client base.  Web services allow a remote client to access functionality (as defined by you, the programmer) via the standard HTTP interface (normally, port:80) to your application and database services.

Back in the day, networking services (semaphores, pipes, streams, message queues, and other forms of IPC) were custom-written and assigned/slaved to unique networking ports for accessing specific service daemons.  Of course, the internet was a kinder, gentler place back then… and a given server may have had dozens, or even hundreds, of non-standard ports open, listening, waiting, for networking service requests.  Or hacking attempts.

Today, web-services is a replacement to dedicated networking apps – all handled by your web server, and all serviced over the same network port: port 80.  Since port 80 is a standard port and, usually, already open on a web-server, additional security risks by opening new non-standard ports for networking services are averted.  Your web-server, such as Apache, now has the responsibility of processing the request and delivering the results to the client.

The web-services component piece is a collection of functions that you’ve written that provide remote clients access to your system.  These functions are accessible only via the web services framework and while they may be duplicated from a dedicated and traditional web-based application, the web-services framework is designed as a stand-alone piece of software.  Think of the web-services piece as your application’s data-processing layer minus the presentation layer.  The “M” and “C” of the “MVC” model.

Initially, when I was tasked with a similar set of objectives, I initially tried xml-rpc.  This led me down a rabbit-hole that spanned nearly a week of my time with the end-result being abandonment.  I hit road-blocks with xml-rpc over server authentication and passing complex objects.  Exacerbating the issue overall is that xml-rpc seems to be dated technology – I had a hard time locating resources that were recent.

Then I stumbled across nuSOAP – it’s free via sourceForge, stable and, using a quote from sourceForge:

“NuSOAP is a rewrite of SOAPx4, provided by NuSphere and Dietrich Ayala. It is a set of PHP classes – no PHP extensions required – that allow developers to create and consume web services based on SOAP 1.1, WSDL 1.1 and HTTP 1.0/1.1.”

nuSOAP seemed to have more of everything available: tutorials, examples, articles, blog posts.  When I started my implementation with nuSOAP, the first thing I received help with was server-level authentication.  I was able to immediately get my remote requests validated by the web-server and handed off to the web-services module!

The major selling point, for me, on nuSOAP is that nuSOAP is self-documenting.  As part of the API functionality, nuSOAP generates HTML pages that documents the exposed services via the WSDL  and also provides you with a complete XSLT definition file!

First off, download and install the nuSOAP libraries – I provided a link to the sourceForge site a couple paragraphs ago – and unpack the tarball.  You’ll end-up with a directory (mine is called: ./nuSOAP) and, within that directory, is the one file you include: nusoap.php.

There are two pieces to this tutorial — a server side piece and a client-side piece.  While you can execute both pieces off the same environment (machine), normally you’d use the client remotely to access the API server-side code.

What’s Not Covered:

Apache.  Apache configuration for your vhost.  Apache .htaccess.  This article assumes that you’ve a working server and that you’re able to install and access the server files via Apache.  Even if your Apache server is a localhost configuration, access the server-side files via localhost client, traversing the TCP stack locally, is still a valid method for testing your web-services server application.

 

Time to push up our sleeves and start working on the server code…

The Web-Services Server

Today, we’re going to write a ping server — where the server has an exposed service (method) named “ping” which takes a single argument (a string) and returns an array back to the calling client.  The return array contains two associative members: a boolean (which should always be true – otherwise there are other issues…) and a string which is the modification of the original string in order to prove that, yes, we went there and we came back.

Because my project, and I’m doing this project for my new company, is going to represent significant effort, size and complexity, I’ve broken out the components of nuSOAP request into exterior files because, later, these will become control file which will, in turn, load files that have been organized into a hierarchy friendly to the application’s data model.

So, if this file, which I’ve named index.php, seems small, remember that you’re not viewing the dependent files (yet).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<pre><?php
/**
 *
 */


// setup...
require('./nuSOAP/nusoap.php');
// set namespace and initiate soap_server instance
$namespace = "http://myapi/index.php";
$server = new soap_server('');
// name the api
$server->configureWSDL("myapi");
// assign namespace
$server->wsdl->schemaTargetNamespace = $namespace;
// register services
require('myServiceRegistrations.php');
// load WSDL...
include('mywsdl.php');
// load services code modules...
require('myServicesModules.php');
// create HHTP listener:
$request = isset($HTTP_RAW_POST_DATA) ? $HTTP_RAW_POST_DATA : '';
$server->service($request);
exit();
?></pre>

So far, so good – let’s take a look at what we’ve just done:

  • we’ve included the nu_soap library…
  • declared our namespace (which is the URL of the api server)
  • instantiated a new soap_server instance and assigned it to the variable $server
  • initialized the WSDL
  • assigned the namespace variable to the WSDL
  • load and register our exposed services
  • load the WSDL
  • load the service code
  • create the HTTP listener
  • invoke the requested service
  • exit

This (index.php) file is the server-side file that will be invoked for ALL future API calls to the service.  It invokes three control files which, in turn loads the services (WSDL definitions), the WSDL variable definitions (think of these as inputs and outputs to your exposed services), and the actual code for all of the services, and their supporting functions, that you’re going to expose via your API.

Side Note:  This is the file you’ll reference in Apache when you’re (preferably) creating a new Virtual Host for the API.  HTTP requests that resolve to your server will be serviced by Apache which will, in turn, serve the results of this program back to the client.

Next, we’re going to define the myServiceRegistrations.php file that is required by index.php.  This file contains the WSDL for each and every exposed service that the API serves.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<pre><?php
/**
 *
 *
 */


$server->register('ping', array('testString' => 'xsd:string'),
                          array('return' => 'tns:aryReturn'),
                  $namespace, false, 'rpc', 'encoded',
'<p><strong>Summary</strong>: returns response to a ping request from the client.  Response
includes testString submitted.  Used to test Server response/connectivity.
</p>
<p>
<strong>Input</strong>: $testString (type: string) and random collection suffices.
</p>
<p>
<strong>Response</strong>: Service returns an array named $aryReturn with two associative
members: &quot;status&quot; and &quot;data&quot;.<br />$aryReturn[&quot;status&quot;] should
<i>always</i> return true.  (A time-out is an implicit false.)<br />If no value was passed
to the service via $testString, then the value of $aryReturn[&quot;data&quot;] will be
empty.<br />Otherwise it will contain the string: &quot;Rcvd: {yourString} (count)&quot; to
show that the message was received and returned. (count) is a character count of the
passed string, also validating that the passed data was received and processed.
</p>
'
);</pre>

This PHP code registers a function called “ping” with the nuSOAP $server instance.  The second parameter is the input to function.  Note that all input (and output) parameters have to be declared as an array even if there’s only a single value being passed.  Also notice that you have to type-cast the variable being passed using XML datatypes.  For your data definitions, you use one of the 44 built-in datatypes defined in this document: http://www.w3.org/TR/2001/REC-xmlschema-2-20010502/.

(For more information on XSD object and XML schema, please visit: http://ws.apache.org/axis/cpp/arch/XSD_Objects.html.)

The third argument to the method “register” is the data type returned.  This is a complex variable called “aryReturn” — don’t worry about right now, we’re going to define this complex-type variable in another configuration file.

The next four arguments are:

  • our $namespace variable we set in index.php
  • boolean false
  • ‘rpc’ for the call type
  • ‘encoded’
Use these values literally.
The last variable is a huge block of HTML.  This block of HTML can be as large, or as small, as you need it to be.  It’s the basis for the WSDL documentation that nuSOAP generates for your client-side developers.
When developers hit the server URL, they’ll be presented with your API documentation that nuSOAP generates from the WSDL file(s).  As your API grows, exposed methods will be listed in the blue-ish box on the left side of the screen.  Clicking on any of the methods will expose the details (requirements) about that method thus:

Nice, huh?

The second file (that we’ve included in our source: (mywsdl.php)) is the WSDL file that defines our data structures that are used as either inputs, outputs, or both, to the exposed services.  Another thing I like about SOAP and nuSOAP is that it introduces a layer of strong-typing to the PHP stack.  You can save a bit of debugging time by requiring that you call a method with EXACTLY this and return EXACTLY that.  Anything else tends to generate a system-level error:

1
2
3
4
<pre>[Thu Sep 22 14:36:59 2011] [error] [client ::1] PHP Fatal error:  Call to a member function addComplexType() on a non-object in /htdocs/LL2/trunk/services/mywsdl.php on line 9
[Thu Sep 22 14:36:59 2011] [error] [client ::1] PHP Stack trace:
[Thu Sep 22 14:36:59 2011] [error] [client ::1] PHP   1. {main}() /htdocs/LL2/trunk/services/ll2wsdl.php:0</pre>
<pre>

This error message, from the apache error log, is somewhat obfuscated in it's meaning.   I attempted to return only the string, by itself, instead of returning the array (of two elements) that I had told nuSOAP I would return for this service.  This error was generated because the types (between the code and the WSDL) of the return variable (structure) did not exactly match.

If you've only ever coded in a loosely-typed language, like PHP, than this part of SOAP is going to be a bit of a ... transition ... for you.  When we say that something, be it a variable, function, or exposed service, is strongly typed, we're declaring the type of that object and, if the type of the object during run-time does not match, then SOAP will force PHP to throw a fatal as shown in the error log snippet above.

Keep this in-mind as you develop exposed services that are increasingly complex.  Since the error messages tend to point you at your code, at the point of failure, it's easy to forget that that the requirements of the underpinnings (in this case, the WSDL), are the root cause of your PHP fatals.

That being said, let's take a look at the WSDL for our ping service:

1
2
3
4
5
6
7
8
9
10
11
<pre><?php
/**
 * WDSL control structures
 *
 * initially, while in dev, this will be one large file but, later, as the product
 * matures, the plan will be to break out the WSDL files into associative files based
 * on the object class being defined.
 */

$server->wsdl->addComplexType('aryReturn', 'complexType', 'struct', 'all', '',
            array('status' => array('name' => 'status', 'type' => 'xsd:boolean'),
                  'data'   => array('name' => 'data',   'type' => 'xsd:string')));</pre>

We're invoking the nuSOAP method addComplexType to define a structure to the WSDL in our Table Name Space (tns).   To do this, we first define the name of the structure that we're going to use: aryReturn and then we define the composition of that structure.

The declaration for this looks a lot like a standard PHP declaration for an array with the exception of the XSD (XML Schema Definition) appended at the end of each element's declaration.  (See the links I embedded above for explanations and examples of valid XSD.)

XSD provides part of the strongly-typed concept for our structure elements.  We're telling nuSOAP to expect a variable structure containing these named elements of this type.

What we have, then, is an associative array with two elements: 'status' and 'data'.  $aryReturn['status'] and $aryReturn['data'] and they're of type BOOL and STRING respectively.

Note, finally, that this variable structure isn't confined to single-use.  Once we've declared it within our tns, it's available to any exposed service where it's needed.  This is the model for my common error structure -- the boolean indicates success or fail on some service operation and the data component contains the relative diagnostic information.

The third and final file we're including into the server source is the code for the exposed service.  This is where you write the function handlers for your services.  Since you've already defined the input parameters, and the return types for the ping service, there's very little left to do.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<pre><?php
/**
 * ping
 *
 * service that confirms server availability
 *
 * input: any string
 *
 * output: reformatted string:  "Rcvd: {string} (charcount{string})"
 *
 * @param string $testString
 * @return string
 */

function ping($testString = '') {
    return(array('status' => true, 'data' => 'Rcvd: ' . $testString . '(' . strlen($testString) . ')'));
}</pre>

Note the following in the code for our exposed service:

  • our exposed method is named ping because that's how we registered the service (myServiceRegistrations.php)
  • we're providing a default type-cast for the input param of string in case the service is invoked without input params.
  • we're returning BOOL true and prepending "Rcvd: " to the received string, and appending a character count to prove that the service successfully responded to the client's request.
  • the return structure exactly matches the WSDL declaration: the name of the array elements, and the element types.
If you've correctly installed and referenced (within your PHP) the nuSOAP libraries, then you should be able to load the url of the new server source file into your web browser to see the nuSOAP-generated documentation for your new web services.  Click on the WSDL service function: ping to see a detailed description of the function.

If you're using IE, then clicking on the WSDL link will return the XML.  If you're using Firefox, Chrome or other browsers, clicking on WSDL will display the generated XML for your service.

Now that the server is working on it's own, it remains fairly useless until we can get a client to connect to it invoke it's methods.  Let's work on the client next...

The Web-Services Client

The web services client application will also be written in PHP.

The web client is an application that connects to remote server using the http port 80.  To do so, you'll need the client to be aware of certain bits of information that may, or may not, be required to access the remote server.

In our client, we're not going to require remote authentication -- but I'll take a quick aside andexplain how you would include this, client-side, if your server required .htaccess authentication.

nuSOAP has a client method called setCredentials which allows you to specify your .htaccess username and password and the authentication schema.  It's a single line of code which is normally used to require not only clients to login to access your API, but, once identified, you can limit the set of exposed methods available to individual clients or groups.

For example, if you have a product you've developed in-house, then you'd want full-access for your front-end web/applications servers.  Your PM later decides to open a subset of the API to the general public and a subscription-based set of exposed methods in order to monetize your product.  Finally, the PM also wants to "white-box" the product so that other companies can use it but with their branding and access to isolated or discrete data sets.

What you'd end-up with is several levels of client access to your web services.  Implementation of limiting exposed services would be handled server-side but it would be based on your client's authentication and possibly the subject of a future tutorial...

So, to the client-side code: (name this file: apiTestClient.php)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<pre><?php
// Pull in the NuSOAP code
require_once('./nuSOAP/nusoap.php');
$proxyhost = isset($_POST['proxyhost']) ? $_POST['proxyhost'] : '';
$proxyport = isset($_POST['proxyport']) ? $_POST['proxyport'] : '';
$proxyusername = isset($_POST['proxyusername']) ? $_POST['proxyusername'] : '';
$proxypassword = isset($_POST['proxypassword']) ? $_POST['proxypassword'] : '';
$useCURL = isset($_POST['usecurl']) ? $_POST['usecurl'] : '0';
$client = new nusoap_client('http://{YOURSERVERURLHERE}/index.php', false, $proxyhost, $proxyport, $proxyusername, $proxypassword);
$err = $client->getError();
if ($err) {
    echo '<h2>Constructor error</h2><pre>' . $err . '</pre>';
}
$client->setUseCurl($useCURL);
$client->useHTTPPersistentConnection();
// Call the SOAP method
$result = $client->call('ping', array('testString' => 'Argle'), 'http://localhost');
// Display the result
if (!$result) {
    echo "service returned false";
} else {
    print_r($result);
}
unset($client);</pre>
<pre>?></pre>

The first thing we do in our client-side code is to include the nuSOAP libraries.

The next five lines of code read from the local POST environment, testing if you've established a proxy for your web-services server and, if so, populating the proxy variables.

The next line instantiates a nuSOAP client and associates it with our remote server.  Change "YOURSERVERURLHERE" to the name of your apache web server URL where you have the server side code installed.  (e.g.: localhost, myserver.com, etc.)

Note the name of the function call: newsoap_client()...as opposed to using the function soapclient().  This function name is legacy-compatible with PHP 5.0's instantiation call:  new soapclient() - the PHP SOAP extension uses the same instantiation function name as the nuSOAP library.  If you have both installed, (PHP 5.0 SOAP extension, and the nuSOAP libraries), executing the client will return errors as you've overloaded the soapclient() function.  (You're calling the PHP SOAP function with the nuSOAP function parameters.)  Rename the soapclient() function to the back-compatible function: newsoap_client().

The next two lines tell the nuSOAP client to useCurl, when possible and if previously saved and, if possible, to use HTTP persistent connections.

Next, we're going to consume the web-services from the server by making a call to the nuSOAP method: call().  The arguments to this method are:

  1. the name of the service being consumed (ping)
  2. the input string (note:  all inputs must be passed as arrays!)
  3. the namespace URI (optional:  WSDL can override)

Store the results of the web-services to the aptly-named variable $results and evaluate it upon return.  If the client call was not successful, then display an error message.

If the client call was successful, then display the contents of the returned array.

Note that you can run this client code from either a browser using the "file://" option or, if the client source code is accessible to apache, then you can display using a browser.

When I run this client-side software in the browser, I get displayed back:

Array ( [status] => 1 [data] => Rcvd: Argle(5) )

So, how do we know that the client went out and successfully returned from the server with the data?  Simple - in the client source, you will not find the literal "Rcvd:" anywhere in the source.  This data was supplied by the server-side function and returned back to the client.  It's a simple example, but it provides proof-of-concept that we're successfully able to connect to a remote web-server and return data created on the remote server back to the client program.

Let's wrap this up...

Summary:

This tutorial (hopefully) explained what web-services are, and provided you with a practical example of a consumable service: ping().  Such a service would normally be invoked as a means of testing server availability.

We created a web-services server file using the nuSOAP library by defining a complex-structure (an associative array) and registering a method with the nuSOAP server.  The method takes an input parameter which, although it's only a single input parameter, must be built and passed as an array construct to the server method.  Next, we showed that by loading the server source into a browser, we learned the nuSOAP provides built-in documentation for your web-services structures and methods.  Which is a very nice-to-have when you're creating your developer documentation!  Next we created the web-services client source code file which connected to our remote server and invoked the server's method: hellow().  If all worked correctly, you displayed the string returned from the remote server within your browser window.

In the next installment, I'll cover the web-services server-side of this business and we'll see how to create the various methods for accessing a mySQL database, complex structures as input and output parameters to those methods, and general debugging techniques.

Thank-you for your patience - I hope this article helped you.

September 26, 2011

Page optimized by WP Minify WordPress Plugin

Our weather forecast is from WP Wunderground