Featured Posts

The 10 Worst Things about World of Warcraft - Mists... I've been playing WoW since vanilla version starting in 2006.  Except for a six-month hiatus in late 2011, I've been a daily player.  I've seen multiple patches come...

Read more

Best Breakfast Burritos, ever! I like eating a good breakfast, usually around lunchtime once I've had my fill of coffee and am awake enough to appreciate a good breakfast. This is my recipe for my ultimate...

Read more

Testing Arrays in PHP - Back to Basics... Sometimes, when you're wallowing through your abstraction class layers, you find yourself using code for simple functions that are normally the focus of an Intro to Programming...

Read more

PHP: Comparing Object Structures I'm working on a project where I am converting an established REST API over to a rabbitMQ service.  Because, you know, dinosaur, I'm continuing to use PHP as my language...

Read more

Mountain Lion and Tunnelblick - Playing Nice Together One of the things that requires some tweaking after the installation of Mac OS X (Mountain Lion) is Tunnelblick, a free and open-source GUI for openVPN.  I use Tunnelblick...

Read more

Subscribe

Mountain Lion and Tunnelblick – Playing Nice Together

Category : Technical
No Gravatar

One of the things that requires some tweaking after the installation of Mac OS X (Mountain Lion) is Tunnelblick, a free and open-source GUI for openVPN.  I use Tunnelblick for work and play so it’s pretty important that it be working correctly.

After Mountain Lion installed, Tunnelblick no longer worked on either of my VPNs.  Scanning the forums, blogs, and support sites, I found several solutions, none of which worked on their own (for me), but instead required combining the solutions to get things to work.

First, you’re going to need to de-install your existing copy of Tunnelblick but, before you do, open a finder window, or cd from the command line, into “~/Library/Application Support/Tunnelblick”.

Copy all of your current configuration files to someplace safe in the event you don’t need new configuration files for your VPN.  Personally, I use BolehVPN and   I generated new key files as part of this process.  My work VPN files are going to require some tweaking too but, either way, it’s always nice to have back-ups so you don’t have to start-over from scratch creating your configs.

I used a sweet little program called AppZapper to remove Tunnelblick – it thoroughly cleans out all files associated with the app – hence the reason why you want to back-up your config files.

Once you’ve removed Tunnelblick, download the beta version of the software.  The current release is versioned at 3.2.6(2891.3007) and was built on May 3, 2012.  This version does not work with Mountain Lion.

You want to download and install Tunnelblick 3.3.  The Beta release version changed four iterations in 24-hours so don’t worry about which rev your getting — this has worked for me since beta release 12.

Once you’re down with the install, you can either download new key files (certificates, keys, and configs with the opvn extension) or you can see if your current configurations will work with the beta.

Once I installed the new key files, I found that I could connect to my VPN provider successfully, but I couldn’t do anything after that point.

If this is the case for you, which you can confirm by loading a website by it’s IP address instead of the URL… As a matter of routine, it’s good to memorize (or at least write-down) this IP address: 74.125.53.100 which is the IP address for google.com.

If you can get Google to load by IP and not by name, (google.com), in the URL bar, then you need to tweak your nameserver settings for your VPN configuration.

Click on the Tunnelblick icon in your menu bar and then click on “VPN Details” at the bottom of the drop down.  This will open a dialog box with your available VPN configurations on the left in a scrolling text box and, on the right, a two-tab  panel for your Logs and Settings.

If your connection failed, scroll through the information under the Log tab and you should see something that looks like this:

2012-07-27 08:51:06 *Tunnelblick process-network-changes: SearchDomains changed from
*
*                     to
*                    <array> {
*                    0 : openvpn
*                    }
*                    pre-VPN was
*

This information shows you that you’ve no search domains selected for the new VPN connection because normally you should see IP address in these containers.  Unless you do everything by IP address, you’re going to be dead in the water…so….

Click on your VPN listing and then click on the Settings tab.

By default, your DNS/WNS configuration has the setting “Set nameserver”.  You want to change this setting to “Set nameserver (3.0b10)”.

Click back to the Log tab and attempt your connection.  You should be able to successfully resolve DNS/WNS at this point however, if for some reason this doesn’t work, then go back to the Settings tab and try the other nameserver options in the drop down.

Finally, something else you may wish to check is your DNS server settings for your OS.  Open System Preferences -> Network -> Advanced… for your internet connection.   Click on the DNS tab and look a your nameservers listed in the box on the left side.

If you’re using your ISP’s router IP as your DNS server, you’re probably going to have a bad time.  I use Google’s nameservers but there are other free nameservers available too, such as OpenDNS.  Whatever you decide to use, enter the nameserver IP address in this block after removing the IP for your ISP or network router.

Google’s DNS nameserver IPs are: 8.8.8.8 and 8.8.4.4.

That’s it!  Hope this helps!

 

 

Web Services with PHP & nuSoap – Part 1.1

Category : Technical
No Gravatar

Introduction

[EDIT] – This is a re-hash of a document I wrote a couple years ago.  There’s been changes to the nuSOAP library and I wanted to document the updates relative to the tutorial series.  Also, I need to fix a lot of the broken-links and code listings since I’ve changes hosting providers since this article was first written.

This article is specific to nuSOAP release 0.9.5 on 2011-01-13.  This tutorial was updated on September 22, 2011.

Once upon a time, I was tasked with developing a web-services by my boss as the integration point between our production application and Sales Force.  At this point, although I’d heard of web services … kind of … didn’t Amazon use web services for something?  Still, I’d never coded a web-services based application before.

At this point, I have to assume you are unfamiliar with the concept of web services and why you may have to create and provide a web-services offering to your client base.  Web services allow a remote client to access functionality (as defined by you, the programmer) via the standard HTTP interface (normally, port:80) to your application and database services.

Back in the day, networking services (semaphores, pipes, streams, message queues, and other forms of IPC) were custom-written and assigned/slaved to unique networking ports for accessing specific service daemons.  Of course, the internet was a kinder, gentler place back then… and a given server may have had dozens, or even hundreds, of non-standard ports open, listening, waiting, for networking service requests.  Or hacking attempts.

Today, web-services is a replacement to dedicated networking apps – all handled by your web server, and all serviced over the same network port: port 80.  Since port 80 is a standard port and, usually, already open on a web-server, additional security risks by opening new non-standard ports for networking services are averted.  Your web-server, such as Apache, now has the responsibility of processing the request and delivering the results to the client.

The web-services component piece is a collection of functions that you’ve written that provide remote clients access to your system.  These functions are accessible only via the web services framework and while they may be duplicated from a dedicated and traditional web-based application, the web-services framework is designed as a stand-alone piece of software.  Think of the web-services piece as your application’s data-processing layer minus the presentation layer.  The “M” and “C” of the “MVC” model.

Initially, when I was tasked with a similar set of objectives, I initially tried xml-rpc.  This led me down a rabbit-hole that spanned nearly a week of my time with the end-result being abandonment.  I hit road-blocks with xml-rpc over server authentication and passing complex objects.  Exacerbating the issue overall is that xml-rpc seems to be dated technology – I had a hard time locating resources that were recent.

Then I stumbled across nuSOAP – it’s free via sourceForge, stable and, using a quote from sourceForge:

“NuSOAP is a rewrite of SOAPx4, provided by NuSphere and Dietrich Ayala. It is a set of PHP classes – no PHP extensions required – that allow developers to create and consume web services based on SOAP 1.1, WSDL 1.1 and HTTP 1.0/1.1.”

nuSOAP seemed to have more of everything available: tutorials, examples, articles, blog posts.  When I started my implementation with nuSOAP, the first thing I received help with was server-level authentication.  I was able to immediately get my remote requests validated by the web-server and handed off to the web-services module!

The major selling point, for me, on nuSOAP is that nuSOAP is self-documenting.  As part of the API functionality, nuSOAP generates HTML pages that documents the exposed services via the WSDL  and also provides you with a complete XSLT definition file!

First off, download and install the nuSOAP libraries – I provided a link to the sourceForge site a couple paragraphs ago – and unpack the tarball.  You’ll end-up with a directory (mine is called: ./nuSOAP) and, within that directory, is the one file you include: nusoap.php.

There are two pieces to this tutorial — a server side piece and a client-side piece.  While you can execute both pieces off the same environment (machine), normally you’d use the client remotely to access the API server-side code.

What’s Not Covered:

Apache.  Apache configuration for your vhost.  Apache .htaccess.  This article assumes that you’ve a working server and that you’re able to install and access the server files via Apache.  Even if your Apache server is a localhost configuration, access the server-side files via localhost client, traversing the TCP stack locally, is still a valid method for testing your web-services server application.

 

Time to push up our sleeves and start working on the server code…

The Web-Services Server

Today, we’re going to write a ping server — where the server has an exposed service (method) named “ping” which takes a single argument (a string) and returns an array back to the calling client.  The return array contains two associative members: a boolean (which should always be true – otherwise there are other issues…) and a string which is the modification of the original string in order to prove that, yes, we went there and we came back.

Because my project, and I’m doing this project for my new company, is going to represent significant effort, size and complexity, I’ve broken out the components of nuSOAP request into exterior files because, later, these will become control file which will, in turn, load files that have been organized into a hierarchy friendly to the application’s data model.

So, if this file, which I’ve named index.php, seems small, remember that you’re not viewing the dependent files (yet).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<pre><?php
/**
 *
 */


// setup...
require('./nuSOAP/nusoap.php');
// set namespace and initiate soap_server instance
$namespace = "http://myapi/index.php";
$server = new soap_server('');
// name the api
$server->configureWSDL("myapi");
// assign namespace
$server->wsdl->schemaTargetNamespace = $namespace;
// register services
require('myServiceRegistrations.php');
// load WSDL...
include('mywsdl.php');
// load services code modules...
require('myServicesModules.php');
// create HHTP listener:
$request = isset($HTTP_RAW_POST_DATA) ? $HTTP_RAW_POST_DATA : '';
$server->service($request);
exit();
?></pre>

So far, so good – let’s take a look at what we’ve just done:

  • we’ve included the nu_soap library…
  • declared our namespace (which is the URL of the api server)
  • instantiated a new soap_server instance and assigned it to the variable $server
  • initialized the WSDL
  • assigned the namespace variable to the WSDL
  • load and register our exposed services
  • load the WSDL
  • load the service code
  • create the HTTP listener
  • invoke the requested service
  • exit

This (index.php) file is the server-side file that will be invoked for ALL future API calls to the service.  It invokes three control files which, in turn loads the services (WSDL definitions), the WSDL variable definitions (think of these as inputs and outputs to your exposed services), and the actual code for all of the services, and their supporting functions, that you’re going to expose via your API.

Side Note:  This is the file you’ll reference in Apache when you’re (preferably) creating a new Virtual Host for the API.  HTTP requests that resolve to your server will be serviced by Apache which will, in turn, serve the results of this program back to the client.

Next, we’re going to define the myServiceRegistrations.php file that is required by index.php.  This file contains the WSDL for each and every exposed service that the API serves.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<pre><?php
/**
 *
 *
 */


$server->register('ping', array('testString' => 'xsd:string'),
                          array('return' => 'tns:aryReturn'),
                  $namespace, false, 'rpc', 'encoded',
'<p><strong>Summary</strong>: returns response to a ping request from the client.  Response
includes testString submitted.  Used to test Server response/connectivity.
</p>
<p>
<strong>Input</strong>: $testString (type: string) and random collection suffices.
</p>
<p>
<strong>Response</strong>: Service returns an array named $aryReturn with two associative
members: &quot;status&quot; and &quot;data&quot;.<br />$aryReturn[&quot;status&quot;] should
<i>always</i> return true.  (A time-out is an implicit false.)<br />If no value was passed
to the service via $testString, then the value of $aryReturn[&quot;data&quot;] will be
empty.<br />Otherwise it will contain the string: &quot;Rcvd: {yourString} (count)&quot; to
show that the message was received and returned. (count) is a character count of the
passed string, also validating that the passed data was received and processed.
</p>
'
);</pre>

This PHP code registers a function called “ping” with the nuSOAP $server instance.  The second parameter is the input to function.  Note that all input (and output) parameters have to be declared as an array even if there’s only a single value being passed.  Also notice that you have to type-cast the variable being passed using XML datatypes.  For your data definitions, you use one of the 44 built-in datatypes defined in this document: http://www.w3.org/TR/2001/REC-xmlschema-2-20010502/.

(For more information on XSD object and XML schema, please visit: http://ws.apache.org/axis/cpp/arch/XSD_Objects.html.)

The third argument to the method “register” is the data type returned.  This is a complex variable called “aryReturn” — don’t worry about right now, we’re going to define this complex-type variable in another configuration file.

The next four arguments are:

  • our $namespace variable we set in index.php
  • boolean false
  • ‘rpc’ for the call type
  • ‘encoded’
Use these values literally.
The last variable is a huge block of HTML.  This block of HTML can be as large, or as small, as you need it to be.  It’s the basis for the WSDL documentation that nuSOAP generates for your client-side developers.
When developers hit the server URL, they’ll be presented with your API documentation that nuSOAP generates from the WSDL file(s).  As your API grows, exposed methods will be listed in the blue-ish box on the left side of the screen.  Clicking on any of the methods will expose the details (requirements) about that method thus:

Nice, huh?

The second file (that we’ve included in our source: (mywsdl.php)) is the WSDL file that defines our data structures that are used as either inputs, outputs, or both, to the exposed services.  Another thing I like about SOAP and nuSOAP is that it introduces a layer of strong-typing to the PHP stack.  You can save a bit of debugging time by requiring that you call a method with EXACTLY this and return EXACTLY that.  Anything else tends to generate a system-level error:

1
2
3
4
<pre>[Thu Sep 22 14:36:59 2011] [error] [client ::1] PHP Fatal error:  Call to a member function addComplexType() on a non-object in /htdocs/LL2/trunk/services/mywsdl.php on line 9
[Thu Sep 22 14:36:59 2011] [error] [client ::1] PHP Stack trace:
[Thu Sep 22 14:36:59 2011] [error] [client ::1] PHP   1. {main}() /htdocs/LL2/trunk/services/ll2wsdl.php:0</pre>
<pre>

This error message, from the apache error log, is somewhat obfuscated in it's meaning.   I attempted to return only the string, by itself, instead of returning the array (of two elements) that I had told nuSOAP I would return for this service.  This error was generated because the types (between the code and the WSDL) of the return variable (structure) did not exactly match.

If you've only ever coded in a loosely-typed language, like PHP, than this part of SOAP is going to be a bit of a ... transition ... for you.  When we say that something, be it a variable, function, or exposed service, is strongly typed, we're declaring the type of that object and, if the type of the object during run-time does not match, then SOAP will force PHP to throw a fatal as shown in the error log snippet above.

Keep this in-mind as you develop exposed services that are increasingly complex.  Since the error messages tend to point you at your code, at the point of failure, it's easy to forget that that the requirements of the underpinnings (in this case, the WSDL), are the root cause of your PHP fatals.

That being said, let's take a look at the WSDL for our ping service:

1
2
3
4
5
6
7
8
9
10
11
<pre><?php
/**
 * WDSL control structures
 *
 * initially, while in dev, this will be one large file but, later, as the product
 * matures, the plan will be to break out the WSDL files into associative files based
 * on the object class being defined.
 */

$server->wsdl->addComplexType('aryReturn', 'complexType', 'struct', 'all', '',
            array('status' => array('name' => 'status', 'type' => 'xsd:boolean'),
                  'data'   => array('name' => 'data',   'type' => 'xsd:string')));</pre>

We're invoking the nuSOAP method addComplexType to define a structure to the WSDL in our Table Name Space (tns).   To do this, we first define the name of the structure that we're going to use: aryReturn and then we define the composition of that structure.

The declaration for this looks a lot like a standard PHP declaration for an array with the exception of the XSD (XML Schema Definition) appended at the end of each element's declaration.  (See the links I embedded above for explanations and examples of valid XSD.)

XSD provides part of the strongly-typed concept for our structure elements.  We're telling nuSOAP to expect a variable structure containing these named elements of this type.

What we have, then, is an associative array with two elements: 'status' and 'data'.  $aryReturn['status'] and $aryReturn['data'] and they're of type BOOL and STRING respectively.

Note, finally, that this variable structure isn't confined to single-use.  Once we've declared it within our tns, it's available to any exposed service where it's needed.  This is the model for my common error structure -- the boolean indicates success or fail on some service operation and the data component contains the relative diagnostic information.

The third and final file we're including into the server source is the code for the exposed service.  This is where you write the function handlers for your services.  Since you've already defined the input parameters, and the return types for the ping service, there's very little left to do.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<pre><?php
/**
 * ping
 *
 * service that confirms server availability
 *
 * input: any string
 *
 * output: reformatted string:  "Rcvd: {string} (charcount{string})"
 *
 * @param string $testString
 * @return string
 */

function ping($testString = '') {
    return(array('status' => true, 'data' => 'Rcvd: ' . $testString . '(' . strlen($testString) . ')'));
}</pre>

Note the following in the code for our exposed service:

  • our exposed method is named ping because that's how we registered the service (myServiceRegistrations.php)
  • we're providing a default type-cast for the input param of string in case the service is invoked without input params.
  • we're returning BOOL true and prepending "Rcvd: " to the received string, and appending a character count to prove that the service successfully responded to the client's request.
  • the return structure exactly matches the WSDL declaration: the name of the array elements, and the element types.
If you've correctly installed and referenced (within your PHP) the nuSOAP libraries, then you should be able to load the url of the new server source file into your web browser to see the nuSOAP-generated documentation for your new web services.  Click on the WSDL service function: ping to see a detailed description of the function.

If you're using IE, then clicking on the WSDL link will return the XML.  If you're using Firefox, Chrome or other browsers, clicking on WSDL will display the generated XML for your service.

Now that the server is working on it's own, it remains fairly useless until we can get a client to connect to it invoke it's methods.  Let's work on the client next...

The Web-Services Client

The web services client application will also be written in PHP.

The web client is an application that connects to remote server using the http port 80.  To do so, you'll need the client to be aware of certain bits of information that may, or may not, be required to access the remote server.

In our client, we're not going to require remote authentication -- but I'll take a quick aside andexplain how you would include this, client-side, if your server required .htaccess authentication.

nuSOAP has a client method called setCredentials which allows you to specify your .htaccess username and password and the authentication schema.  It's a single line of code which is normally used to require not only clients to login to access your API, but, once identified, you can limit the set of exposed methods available to individual clients or groups.

For example, if you have a product you've developed in-house, then you'd want full-access for your front-end web/applications servers.  Your PM later decides to open a subset of the API to the general public and a subscription-based set of exposed methods in order to monetize your product.  Finally, the PM also wants to "white-box" the product so that other companies can use it but with their branding and access to isolated or discrete data sets.

What you'd end-up with is several levels of client access to your web services.  Implementation of limiting exposed services would be handled server-side but it would be based on your client's authentication and possibly the subject of a future tutorial...

So, to the client-side code: (name this file: apiTestClient.php)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<pre><?php
// Pull in the NuSOAP code
require_once('./nuSOAP/nusoap.php');
$proxyhost = isset($_POST['proxyhost']) ? $_POST['proxyhost'] : '';
$proxyport = isset($_POST['proxyport']) ? $_POST['proxyport'] : '';
$proxyusername = isset($_POST['proxyusername']) ? $_POST['proxyusername'] : '';
$proxypassword = isset($_POST['proxypassword']) ? $_POST['proxypassword'] : '';
$useCURL = isset($_POST['usecurl']) ? $_POST['usecurl'] : '0';
$client = new nusoap_client('http://{YOURSERVERURLHERE}/index.php', false, $proxyhost, $proxyport, $proxyusername, $proxypassword);
$err = $client->getError();
if ($err) {
    echo '<h2>Constructor error</h2><pre>' . $err . '</pre>';
}
$client->setUseCurl($useCURL);
$client->useHTTPPersistentConnection();
// Call the SOAP method
$result = $client->call('ping', array('testString' => 'Argle'), 'http://localhost');
// Display the result
if (!$result) {
    echo "service returned false";
} else {
    print_r($result);
}
unset($client);</pre>
<pre>?></pre>

The first thing we do in our client-side code is to include the nuSOAP libraries.

The next five lines of code read from the local POST environment, testing if you've established a proxy for your web-services server and, if so, populating the proxy variables.

The next line instantiates a nuSOAP client and associates it with our remote server.  Change "YOURSERVERURLHERE" to the name of your apache web server URL where you have the server side code installed.  (e.g.: localhost, myserver.com, etc.)

Note the name of the function call: newsoap_client()...as opposed to using the function soapclient().  This function name is legacy-compatible with PHP 5.0's instantiation call:  new soapclient() - the PHP SOAP extension uses the same instantiation function name as the nuSOAP library.  If you have both installed, (PHP 5.0 SOAP extension, and the nuSOAP libraries), executing the client will return errors as you've overloaded the soapclient() function.  (You're calling the PHP SOAP function with the nuSOAP function parameters.)  Rename the soapclient() function to the back-compatible function: newsoap_client().

The next two lines tell the nuSOAP client to useCurl, when possible and if previously saved and, if possible, to use HTTP persistent connections.

Next, we're going to consume the web-services from the server by making a call to the nuSOAP method: call().  The arguments to this method are:

  1. the name of the service being consumed (ping)
  2. the input string (note:  all inputs must be passed as arrays!)
  3. the namespace URI (optional:  WSDL can override)

Store the results of the web-services to the aptly-named variable $results and evaluate it upon return.  If the client call was not successful, then display an error message.

If the client call was successful, then display the contents of the returned array.

Note that you can run this client code from either a browser using the "file://" option or, if the client source code is accessible to apache, then you can display using a browser.

When I run this client-side software in the browser, I get displayed back:

Array ( [status] => 1 [data] => Rcvd: Argle(5) )

So, how do we know that the client went out and successfully returned from the server with the data?  Simple - in the client source, you will not find the literal "Rcvd:" anywhere in the source.  This data was supplied by the server-side function and returned back to the client.  It's a simple example, but it provides proof-of-concept that we're successfully able to connect to a remote web-server and return data created on the remote server back to the client program.

Let's wrap this up...

Summary:

This tutorial (hopefully) explained what web-services are, and provided you with a practical example of a consumable service: ping().  Such a service would normally be invoked as a means of testing server availability.

We created a web-services server file using the nuSOAP library by defining a complex-structure (an associative array) and registering a method with the nuSOAP server.  The method takes an input parameter which, although it's only a single input parameter, must be built and passed as an array construct to the server method.  Next, we showed that by loading the server source into a browser, we learned the nuSOAP provides built-in documentation for your web-services structures and methods.  Which is a very nice-to-have when you're creating your developer documentation!  Next we created the web-services client source code file which connected to our remote server and invoked the server's method: hellow().  If all worked correctly, you displayed the string returned from the remote server within your browser window.

In the next installment, I'll cover the web-services server-side of this business and we'll see how to create the various methods for accessing a mySQL database, complex structures as input and output parameters to those methods, and general debugging techniques.

Thank-you for your patience - I hope this article helped you.

September 26, 2011

Secure Access to Cloud-based Source-Code

Category : Technical
No Gravatar

I’ve had this idea for about a week now — I want to store my working source-code tree in the cloud, securely, so that I can access it from my machine at work, or from home.  I use a laptop at work as my primary machine — which is cool — but I really hate lugging the damn thing back-and-forth from work to home.

In my mind, it introduces risk – shoving a laptop into a backpack and trundling 70 or so miles (one-way) just doesn’t seem, to me, to be the best way to treat delicate electronics, even if said device was designed to be portable.  There’s also the additional liability of theft or loss of the device.

Like most geeks, my home machine makes a far better development environment because it has significantly more display real-estate, more memory, and faster everything else.  I don’t have to hunt for a spare power receptacle to plug the laptop into, or work off the kitchen table because my desk is already at capacity.

So I came up with the idea (and I’m not claiming this to be original – but it does work) that if I could store my source code in the cloud, then all I’d need is a duplicate operating environment (apache, mysql and the db contents, etc.) while I ran my development source from the cloud, pushing it to the stage-server when necessary, thereby always maintaining the code in a consistent state across platforms.

I need the repository to be stored under subversion, and I want really decent encryption so that if the account gets hacked, my code isn’t exposed.  (Protect corporate assets.)

Oh, and I want it to be free.   :-)

And to be large enough to store my entire project.  (I like CloudApp and DropBox, but I don’t feel they offer either enough space for what I need to do, or the ability to access the remote “device” as a filesystem.)  Here we go…

Adding DBNavigator tool to PHPStorm IDE…

Category : Technical
No Gravatar

I love PHPStorm as my IDE of choice — this replaced my Zend CE IDE last year following the PHP-Conference in Santa Clara which I was fortunate enough to be able to attend.  Yesterday, I discovered add-ons, or in the PHPStorm nomenclature, Plug-ins.  Plug-ins are programs which integrate into the IDE with the intent and purpose of making your life a lot easier.

Initially, what caught my eye was a plug-in called “CSS-X-Fire” which integrates the Firebug CSS editor into the IDE with CSS capabilities.  Since I’ve been debugging my fair share of CSS lately, I thought this would be a pretty cool tool to have at my disposal.  Let’s see what else they have…

*gasp*  SO MUCH COOL STUFF! *gasp*

I installed the BASH shell next just for fun which not only gives me a shell within my IDE, but also provides BASH syntax highlighting.  And, yes, my work-site I support has a lot of BASH utility scripts, so cool.

Next, I see a gem called Database Navigator written by Mr. Dan Cioca.  This plug-in gives you access to your MySQL environment, through the IDE.

Covering Indexes in mySQL…

Category : Technical
No Gravatar

This is going to be a quick-and-dirty discussion on the importance of covering indexes using mySQL.  I recently was hired to steward a LAMP system that’s fairly complex.  It relies heavily on mySQL as the db-engine and has mostly PHP driving the business logic within a Smarty template engine.

So far, by adjusting tunables within mySQL, I’ve been able to improve response times (world-wide page load averages) by about 25%.  Now it’s time to start ploughing through the slow-query logs since the db server is stable under the current loads and we’ve established a zero-state, through metrics, of baseline performance.

A db index, or indexing to describe the act of adding one or more indexes to a table or tables, to seek improvements in data access and retrieval, is often seen as something akin to the burning of incense and killing of goats when scrying the fortune of the gods.  As in all things performance-related to mySQL, it requires an continuous cycle of minute adjustment, measurement, and evaluation.

We’re going to talk about covering indexes today — (here’s a good overview of indexes courtesy of wikipedia — good to know, or good review!) — which are indexes that contain all of the data from the query or, in other words, the data is stored with the index and is returned by accessing the index without doing a table look-up or scan.

Searching embedded array structures in Mongodb documents…

Category : Technical
No Gravatar

I am working with a fairly complex data structure, which I am storing in mongodb.  The document, as these are known in the parlance of Mongo, contains six embedded documents, one of which is an array of documents.

It is this document, the array, which I am interested in searching.

The mongo home page is a great resource for us mongo-nubs — the documentation available there is well-written, clear, and concise.  Sometimes, I can even find what I need, before I even realize that I need it!

However, like most documentation, and this is the former support-engineer in me talking, developer documentation is written in a tone of a “solution” and not as a “problem”.  In other words, when I’m looking to solve a problem I am having, I am searching in the terms of my problem.  Once I’ve solved the problem, I will usually document what I’ve learned — the challenge is to not write the document using the vocabulary of the solution, but to use the vocabulary of the problem (seeking) instead.

Anyway, the problem statement for this entry is to be able to search a field within an array of fields within an embedded document, within a mongodb document.  Here we go….

Fatal Error: Class ‘Mongo’ Not Found…

Category : Technical
No Gravatar

Working on this Mongo script in PHP – the script is meant to be run from the command line as there’s about seven different parameters I need to pass into it.  The script itself takes a defined CSV file and build a multi-collection structure for storing in  as a Mongo table.

I get the basic stuff working – the data is porting over and storing very nicely into Mongo — and I go to work on the command-line options (argc/argv stuff) for the script.  First thing I do, is specify the php executable to the shell:

1
2
#! /usr/bin/php
<?php

And I go on to code the handlers for the command-line arguments on a Friday.  When I come back to work on Monday, I go back to run the script and I get the error:

Fatal Error: Class Mongo not found in {source code} on Line such-and-such… which is a huge WTF to me because we’re talking about  nothing changing in the source code over the weekend…

I run mac ports and get a PHP update but I get the same error.

I check that the mongo extensions are loaded with php -i:

mongo
MongoDB Support => enabled
Version => 1.0.9
Directive => Local Value => Master Value
mongo.allow_persistent => 1 => 1
mongo.auto_reconnect => 1 => 1
mongo.chunk_size => 262144 => 262144
mongo.cmd => $ => $
mongo.default_host => localhost => localhost
mongo.default_port => 27017 => 27017
mongo.long_as_object => 0 => 0
mongo.native_long => 0 => 0
mongo.utf8 => 1 => 1

I google the error and read that (derp-derp) the mongo extension is probably not loaded in the php-cli.ini file.  Which I search for but don’t seem to have on my system.

I do have the Zend CE install still hanging around and since I’m using PHPStorm now (it rocks!), I nuke the Zend installation.  Nope – still getting the error.  I run it past a couple of the Mongo/PHP wonks here at work and get lots of hmmm….but it’s a head-scratcher for all of us.

Then, right before I fall asleep last night, I get the answer…it’s in the shell directive in the script…the /usr/bin/php is not the same as the mac-ports version of php.  I ls -l and file the two php files and I get:

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica} p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px} span.Apple-tab-span {white-space:pre}

$ ls -l /usr/bin/php

-rwxr-xr-x  1 root  wheel  30168960 Aug 22 19:44 /usr/bin/php

$ file /usr/bin/php

/usr/bin/php: Mach-O universal binary with 3 architectures

/usr/bin/php (for architecture x86_64): Mach-O 64-bit executable x86_64

/usr/bin/php (for architecture i386): Mach-O executable i386

$ ls -l /opt/local/bin/php

-rwxr-xr-x  2 root  admin  6054056 Jan 10 14:49 /opt/local/bin/php

$ file /opt/local/bin/php

/opt/local/bin/php: Mach-O 64-bit executable x86_64

Once I change the directive in the script source to point the shell to the correct executable, the mac-ports version of PHP, my Mongo-Fatal error goes away and life, once again, is good.

MongoCursorException: E11000 duplicate key error index

Category : Technical
No Gravatar

So I’m working on a project where I’m taking a csv file that contains a little more than 100 columns of data by 10,000 rows.  (It’s a sample db file — the final file will be about 200,000,000 rows…) and writing a PHP script to process the csv file into structures that can be inserted as mongodb collections.

I’m rocking along and all is working well for initial tests of the algorithm (header + first row of actual data) but when I turn-on processing for the other 9,999 rows, all I get stored into mongo is the first row of data.

I add an echo statement after the insert and I see 10,000 names scroll across my terminal.  So the problem isn’t that I’m not getting the data, it’s that the data isn’t being stored into Mongo.  I try turning on safe writes on my $mongo->insert() function and *bam*, error message:

Page optimized by WP Minify WordPress Plugin

The forecast for 92143 by Wordpress Weather