Ajaxonomy Adds OpenID Support

If you have been using our site recently you may have noticed that the site looks different. We have redesigned the site to be more integrated with our social features. Among the new features you will notice are a buddy list, private messaging, each user has a blog, user profiles, a new comment system and more.

One of the changes that I am most happy about is that we now support OpenID. If you already have an OpenID you can use it to login for all of the above mentioned features. Soon you will be able to use your OpenID to login to sites such as Digg, so you will probably want to get one if you don't already have one.

As always we look forward to getting your feedback about the new features. And don't forget to use your OpenID to login (although you can login without an OpenID)!

Cross-Domain XML Access - a.k.a. Server Side Proxy

Tagged:  

In developing Ajax applications there are quite a few times that you may need to consume some type of XML from a different domain (especially in the case of mashups). Since retrieving XML from a different domain through the browser is a security restriction you have to find another way of getting to the XML. One way to get around this is to use JSON data however many API's are only available in XML, so it is very possible that XML will be your only option.

In order to get around these security restrictions we need to use a server side proxy. How this works is that the server retrieves the XML from the other domain (which is not stopped by security restrictions) and then returns it to the Ajax application using the XMHTTPRequest object. This tricks the browser into thinking that the XML comes from the same domain.

So without further delay below is some PHP code that I have used to create a server side proxy. You just need to pass in the URL using the proxy_url variable that is passed to the PHP through the URL.

<?php
//          FILE: proxy.php
//
// LAST MODIFIED: 2006-03-23
//
//        AUTHOR: Troy Wolf 
//
//   DESCRIPTION: Allow scripts to request content they otherwise may not be
//                able to. For example, AJAX (XmlHttpRequest) requests from a
//                client script are only allowed to make requests to the same
//                host that the script is served from. This is to prevent
//                "cross-domain" scripting. With proxy.php, the javascript
//                client can pass the requested URL in and get back the
//                response from the external server.
//
//         USAGE: "proxy_url" required parameter. For example:
//                http://www.mydomain.com/proxy.php?proxy_url=http://www.yahoo.com
//

// proxy.php requires Troy's class_http. http://www.troywolf.com/articles
// Alter the path according to your environment.
require_once("class_http.php");

$proxy_url = isset($_GET['proxy_url'])?$_GET['proxy_url']:false;
if (!$proxy_url) {
    header("HTTP/1.0 400 Bad Request");
    echo "proxy.php failed because proxy_url parameter is missing";
    exit();
}

// Instantiate the http object used to make the web requests.
// More info about this object at www.troywolf.com/articles
if (!$h = new http()) {
    header("HTTP/1.0 501 Script Error");
    echo "proxy.php failed trying to initialize the http object";
    exit();
}

$h->url = $proxy_url;
$h->postvars = $_POST;
if (!$h->fetch($h->url)) {
    header("HTTP/1.0 501 Script Error");
    echo "proxy.php had an error attempting to query the url";
    exit();
}

// Forward the headers to the client.
$ary_headers = split("\n", $h->header);
foreach($ary_headers as $hdr) { header($hdr); }

// Send the response body to the client.
echo $h->body;
?>

Below is the class_http that is referenced in the above.

<?php
/*
* Filename.......: class_http.php
* Author.........: Troy Wolf [troy@troywolf.com]
* Last Modified..: Date: 2006/03/06 10:15:00
* Description....: Screen-scraping class with caching. Includes image_cache.php
                   companion script. Includes static methods to extract data
                   out of HTML tables into arrays or XML. Now supports sending
                   XML requests and custom verbs with support for making
                   WebDAV requests to Microsoft Exchange Server.
*/

class http {
    var $log;
    var $dir;
    var $name;
    var $filename;
    var $url;
    var $port;
    var $verb;
    var $status;
    var $header;
    var $body;
    var $ttl;
    var $headers;
    var $postvars;
    var $xmlrequest;
    var $connect_timeout;
    var $data_ts;
    
    /*
    The class constructor. Configure defaults.
    */
    function http() {
        $this->log = "New http() object instantiated.<br />\n";
        
        /*
        Seconds to attempt socket connection before giving up.
        */
        $this->connect_timeout = 30;
        
        /*
        Set the 'dir' property to the directory where you want to store the cached
        content. I suggest a folder that is not web-accessible.
        End this value with a "/".
        */
        $this->dir = realpath("./")."/"; //Default to current dir.

        $this->clean();               

        return true;
    }
    
    /*
    fetch() method to get the content. fetch() will use 'ttl' property to
    determine whether to get the content from the url or the cache.
    */
    function fetch($url="", $ttl=0, $name="", $user="", $pwd="", $verb="GET") {
        $this->log .= "--------------------------------<br />fetch() called
\n"; $this->log .= "url: ".$url."<br />\n"; $this->status = ""; $this->header = ""; $this->body = ""; if (!$url) { $this->log .= "OOPS: You need to pass a URL!<br />"; return false; } $this->url = $url; $this->ttl = $ttl; $this->name = $name; $need_to_save = false; if ($this->ttl == "0") { if (!$fh = $this->getFromUrl($url, $user, $pwd, $verb)) { return false; } } else { if (strlen(trim($this->name)) == 0) { $this->name = MD5($url); } $this->filename = $this->dir."http_".$this->name; $this->log .= "Filename: ".$this->filename."<br />"; $this->getFile_ts(); if ($this->ttl == "daily") { if (date('Y-m-d',$this->data_ts) != date('Y-m-d',time())) { $this->log .= "cache has expired<br />"; if (!$fh = $this->getFromUrl($url, $user, $pwd, $verb)) { return false; } $need_to_save = true; if ($this->getFromUrl()) { return $this->saveToCache(); } } else { if (!$fh = $this->getFromCache()) { return false; } } } else { if ((time() - $this->data_ts) >= $this->ttl) { $this->log .= "cache has expired<br />"; if (!$fh = $this->getFromUrl($url, $user, $pwd, $verb)) { return false; } $need_to_save = true; } else { if (!$fh = $this->getFromCache()) { return false; } } } } /* Get response header. */ $this->header = fgets($fh, 1024); $this->status = substr($this->header,9,3); while ((trim($line = fgets($fh, 1024)) != "") && (!feof($fh))) { $this->header .= $line; if ($this->status=="401" and strpos($line,"WWW-Authenticate: Basic realm=\"")===0) { fclose($fh); $this->log .= "Could not authenticate<br />\n"; return FALSE; } } /* Get response body. */ while (!feof($fh)) { $this->body .= fgets($fh, 1024); } fclose($fh); if ($need_to_save) { $this->saveToCache(); } return $this->status; } /* PRIVATE getFromUrl() method to scrape content from url. */ function getFromUrl($url, $user="", $pwd="", $verb="GET") { $this->log .= "getFromUrl() called<br />"; preg_match("~([a-z]*://)?([^:^/]*)(:([0-9]{1,5}))?(/.*)?~i", $url, $parts); $protocol = $parts[1]; $server = $parts[2]; $port = $parts[4]; $path = $parts[5]; if ($port == "") { if (strtolower($protocol) == "https://") { $port = "443"; } else { $port = "80"; } } if ($path == "") { $path = "/"; } if (!$sock = @fsockopen(((strtolower($protocol) == "https://")?"ssl://":"").$server, $port, $errno, $errstr, $this->connect_timeout)) { $this->log .= "Could not open connection. Error " .$errno.": ".$errstr."<br />\n"; return false; } $this->headers["Host"] = $server.":".$port; if ($user != "" && $pwd != "") { $this->log .= "Authentication will be attempted<br />\n"; $this->headers["Authorization"] = "Basic ".base64_encode($user.":".$pwd); } if (count($this->postvars) > 0) { $this->log .= "Variables will be POSTed<br />\n"; $request = "POST ".$path." HTTP/1.0\r\n"; $post_string = ""; foreach ($this->postvars as $key=>$value) { $post_string .= "&".urlencode($key)."=".urlencode($value); } $post_string = substr($post_string,1); $this->headers["Content-Type"] = "application/x-www-form-urlencoded"; $this->headers["Content-Length"] = strlen($post_string); } elseif (strlen($this->xmlrequest) > 0) { $this->log .= "XML request will be sent<br />\n"; $request = $verb." ".$path." HTTP/1.0\r\n"; $this->headers["Content-Length"] = strlen($this->xmlrequest); } else { $request = $verb." ".$path." HTTP/1.0\r\n"; } #echo "<br />request: ".$request; if (fwrite($sock, $request) === FALSE) { fclose($sock); $this->log .= "Error writing request type to socket<br />\n"; return false; } foreach ($this->headers as $key=>$value) { if (fwrite($sock, $key.": ".$value."\r\n") === FALSE) { fclose($sock); $this->log .= "Error writing headers to socket<br />\n"; return false; } } if (fwrite($sock, "\r\n") === FALSE) { fclose($sock); $this->log .= "Error writing end-of-line to socket<br />\n"; return false; } #echo "
post_string: ".$post_string; if (count($this->postvars) > 0) { if (fwrite($sock, $post_string."\r\n") === FALSE) { fclose($sock); $this->log .= "Error writing POST string to socket<br />\n"; return false; } } elseif (strlen($this->xmlrequest) > 0) { if (fwrite($sock, $this->xmlrequest."\r\n") === FALSE) { fclose($sock); $this->log .= "Error writing xml request string to socket<br />\n"; return false; } } return $sock; } /* PRIVATE clean() method to reset the instance back to mostly new state. */ function clean() { $this->status = ""; $this->header = ""; $this->body = ""; $this->headers = array(); $this->postvars = array(); /* Try to use user agent of the user making this request. If not available, default to IE6.0 on WinXP, SP1. */ if (isset($_SERVER['HTTP_USER_AGENT'])) { $this->headers["User-Agent"] = $_SERVER['HTTP_USER_AGENT']; } else { $this->headers["User-Agent"] = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"; } /* Set referrer to the current script since in essence, it is the referring page. */ if (substr($_SERVER['SERVER_PROTOCOL'],0,5) == "HTTPS") { $this->headers["Referer"] = "https://".$_SERVER['HTTP_HOST'].$_SERVER['REQUEST_URI']; } else { $this->headers["Referer"] = "http://".$_SERVER['HTTP_HOST'].$_SERVER['REQUEST_URI']; } } /* PRIVATE getFromCache() method to retrieve content from cache file. */ function getFromCache() { $this->log .= "getFromCache() called<br />"; //create file pointer if (!$fp=@fopen($this->filename,"r")) { $this->log .= "Could not open ".$this->filename."<br />"; return false; } return $fp; } /* PRIVATE saveToCache() method to save content to cache file. */ function saveToCache() { $this->log .= "saveToCache() called<br />"; //create file pointer if (!$fp=@fopen($this->filename,"w")) { $this->log .= "Could not open ".$this->filename."<br />"; return false; } //write to file if (!@fwrite($fp,$this->header."\r\n".$this->body)) { $this->log .= "Could not write to ".$this->filename."<br />"; fclose($fp); return false; } //close file pointer fclose($fp); return true; } /* PRIVATE getFile_ts() method to get cache file modified date. */ function getFile_ts() { $this->log .= "getFile_ts() called<br />"; if (!file_exists($this->filename)) { $this->data_ts = 0; $this->log .= $this->filename." does not exist<br />"; return false; } $this->data_ts = filemtime($this->filename); return true; } /* Static method table_into_array() Generic function to return data array from HTML table data rawHTML: the page source needle: optional string to start parsing source from needle_within: 0 = needle is BEFORE table, 1 = needle is within table allowed_tags: list of tags to NOT strip from data, e.g. "<a><b>" */ function table_into_array($rawHTML,$needle="",$needle_within=0,$allowed_tags="") { $upperHTML = strtoupper($rawHTML); $idx = 0; if (strlen($needle) > 0) { $needle = strtoupper($needle); $idx = strpos($upperHTML,$needle); if ($idx === false) { return false; } if ($needle_within == 1) { $cnt = 0; while(($cnt < 100) && (substr($upperHTML,$idx,6) != "<TABLE")) { $idx = strrpos(substr($upperHTML,0,$idx-1),"<"); $cnt++; } } } $aryData = array(); $rowIdx = 0; /* If this table has a header row, it may use TD or TH, so check special for this first row. */ $tmp = strpos($upperHTML,"<TR",$idx); if ($tmp === false) { return false; } $tmp2 = strpos($upperHTML,"</TR>",$tmp); if ($tmp2 === false) { return false; } $row = substr($rawHTML,$tmp,$tmp2-$tmp); $pattern = "/<TH>|<TH\ |<TD>|<TD\ /"; preg_match($pattern,strtoupper($row),$matches); $hdrTag = $matches[0]; while ($tmp = strpos(strtoupper($row),$hdrTag) !== false) { $tmp = strpos(strtoupper($row),">",$tmp); if ($tmp === false) { return false; } $tmp++; $tmp2 = strpos(strtoupper($row),"</T"); $aryData[$rowIdx][] = trim(strip_tags(substr($row,$tmp,$tmp2-$tmp),$allowed_tags)); $row = substr($row,$tmp2+5); preg_match($pattern,strtoupper($row),$matches); $hdrTag = $matches[0]; } $idx = strpos($upperHTML,"</TR>",$idx)+5; $rowIdx++; /* Now parse the rest of the rows. */ $tmp = strpos($upperHTML,"<<R",$idx); if ($tmp === false) { return false; } $tmp2 = strpos($upperHTML,"</TABLE>",$idx); if ($tmp2 === false) { return false; } $table = substr($rawHTML,$tmp,$tmp2-$tmp); while ($tmp = strpos(strtoupper($table),"<TR") !== false) { $tmp2 = strpos(strtoupper($table),"</TR"); if ($tmp2 === false) { return false; } $row = substr($table,$tmp,$tmp2-$tmp); while ($tmp = strpos(strtoupper($row),"<TD") !== false) { $tmp = strpos(strtoupper($row),">",$tmp); if ($tmp === false) { return false; } $tmp++; $tmp2 = strpos(strtoupper($row),"</TD"); $aryData[$rowIdx][] = trim(strip_tags(substr($row,$tmp,$tmp2-$tmp),$allowed_tags)); $row = substr($row,$tmp2+5); } $table = substr($table,strpos(strtoupper($table),"</TR>")+5); $rowIdx++; } return $aryData; } /* Static method table_into_xml() Generic function to return xml dataset from HTML table data rawHTML: the page source needle: optional string to start parsing source from allowedTags: list of tags to NOT strip from data, e.g. "<a><b>" */ function table_into_xml($rawHTML,$needle="",$needle_within=0,$allowedTags="") { if (!$aryTable = http::table_into_array($rawHTML,$needle,$needle_within,$allowedTags)) { return false; } $xml = "<?xml version=\"1.0\" standalone=\"yes\" \?\>\n"; $xml .= "<TABLE>\n"; $rowIdx = 0; foreach ($aryTable as $row) { $xml .= "\t<ROW id=\"".$rowIdx."\">\n"; $colIdx = 0; foreach ($row as $col) { $xml .= "\t\t<COL id=\"".$colIdx."\">".trim(utf8_encode(htmlspecialchars($col)))."</COL>\n"; $colIdx++; } $xml .= "\t</ROW>\n"; $rowIdx++; } $xml .= "</TABLE>"; return $xml; } } ?>

Click here to be taken to the page where you can download all of the code above.

So now you have some code to create a server side proxy and can use it to create a great new mashup! If you know of any other good server side proxy code in PHP, Ruby, Java, .Net or any other web language please leave it in the comments.

JavaScript Game Library - Start Playing Your Games

Tagged:  

Well it is the weekend and time for some fun. I have found a good JavaScript Game Library. The library has some good 2-D gaming functionality. Including sprite animation (including properties for jumping and falling) tile handling for backgrounds and platforms.

Click here to go to the page where you can download the library.

There are also quite a few fun games that have been built with the library. Click here to go to the games.

Now that you have a JavaScript library for game creation you can extend it using Ajax (I would recommend the use of JSON instead of XML) so that you could make a network game. If you create a game let me know about it. I would love to play it.

videogamepic.jpg

ajax im 3.2 released

Tagged:  

Last month we had a post on the ajax chat and instant messaging clients created by unwieldly studios.

ajax im logo

I've just been notified that a new version their instant messenger ajax im 3.2 has been released, and there are many changes including:

  • a major overhaul of the code: everything (PHP and JS) is now
    object-oriented instead of procedural
  • multiple language support
  • admin panel added, supports searching for users, banning, kicking,
    and making/removing admin
  • PHP-based sessions implemented, so the username and password isn't
    sent on every message request
  • and many others.

ajax im buddy list

You can read more about ajax im at ajaxim.com and you can view a demo at ajaxim.net

Demo accounts are usernames "test", and "test[1 to 4]". Password is "test".

Making Prototype Smaller

Tagged:  

Prototype is one of the most used JavaScript libraries and is used for many other libraries. The problem with Prototype is that it is a rather large library. This has forced many people to either shy away from the library or to try and modify it. I say that just for the $ (for those of you who haven't use prototype the $ is used like document.getElementById) functionality the library is worth using.

I ran Prototype through ShrinkSafe and was able to get the code quite a bit smaller. The Prototype.js (I'm using version 1.6) file went from 122K down to 82K. That is an amazing 40K less which helps in using the library.

Below is a demo using the compressed and uncompressed versions of Prototype (the demo uses Script.aculo.us 1.8 and is taken from the demos that come with that library)
Click here to see the uncompressed version
Click here to see the compressed version

You can download the compressed version of Prototype here.

Now that you have a smaller version of Prototype use it to improve the load time of your application.

prototype.png

Developing Google Web Toolkit Applications with Netbeans 6

Tagged:  

With the recent release of Netbeans 6 there is a lot of interesting things happening in the Java world. One of the best toolkits for making Ajax applications for Java developers is the Google Web Toolkit. In case you have never used the Google Web Toolkit (a.k.a GWT) it makes it much easier for Java programmers to create Ajax applications as you can code JavaScript in Java. Netbeans 6 now has a GWT plug-in to help you develop your GWT based applications in NetBeans combining the power of these great tools.

Below is an excerpt that will get you started in your development.

Although GWT is not supported in NetBeans 6 out of the box, you can download this GWT plug-in and start developing GWT-based applications in NetBeans.

The first step is to install the plug-in using the Plug-in manager. Go to the "Tools | Plugins" menu action, switch to the "Downloaded" tab and locate the plug-in on your disk drive. You don't even have to restart your IDE - GWT support is instantly available for you!

The plug-in is seamlessly integrated into NetBeans IDE. That means that when you create a new web application GWT is shown as one of the available frameworks in the last step of the New project wizard. Here you can specify the GWT installation folder and choose the main entry point class of your GWT module.

newwebapp.jpg

You can use the plug-in both if you start from scratch or if you want to work on an existing GWT application. So if you used a different IDE than NetBeans before, it is easy to switch the GWT application to NetBeans. You just point the wizard to your existing GWT application and create a new project from existing sources.

Once you get the project created you can run the application simply by hitting the Run button. There are two options – you can either use the default Run action which deploys the application to the application server and opens your default web browser. The other option is to run GWT in hosted mode and then the GWT development shell is opened and you can see your application inside of it.

run.jpg

Debugging is also supported, so you can just run the Debug action and then the runtime is ran in debug mode. You can simply add breakpoints, step into, step out, etc. as you would expect in a regular web application.

debugging.jpg

NetBeans already provides lots of tooling out of the box that you can take advantage of, like a powerful XML editor, HTML editor and of course a Java editor with code completion and quick fixes. NetBeans 6 made huge strides in improving the editing experience and it shows when developing GWT applications, too. All GWT APIs are available for you including javadoc documentation, because the GWT jars get automatically added into the project during it's creation.

To learn more about GWT support in NetBeans, the project homepage and screencast can help you get started. Sang Shin, a member of the Sun technology evangelism team, also created a free course for GWT and NetBeans, so you can learn from his examples and do the hands-on lab.

The plug-in was developed as an open source project so we encourage developers to join the project and contribute. There are many ways you can contribute, even submitting an issue or request for enhancement counts.

The future roadmap contains exciting features such as refactoring for GWT and GWT-specific quick fixes in the editor which will make developing GWT code even more comfortable. We are always looking for feedback, so if you try out the project let the developers know what you think.

Click here to read the full post.

Now that you know about these great tools get to coding some great Ajax applications in Java using the Google Web Toolkit!

DWR 3.0 vision

DWR 3.0 is going to be released soon. Following are vision from Joe, Founder of DWR

DWR 2.0 has been out for 6 months or so. At the time, I swore that the next release would be a small one, called 2.1. However it appears that I’m not good at swearing because there is lots in the next release - I think we’re going to have to call it 3.0.

Since 2.0, we've been working on the following adding support for JSON, Bayeux, images/binary file upload/download, a Hub with JMS/OAA support and more reverse ajax APIs. I also want to get some Gears integration going.

There are also a whole set of non-functional things to consider:
* Moving the website to directwebremoting.org
* Restart chasing CLAs, using a foundation CLA rather than a Getahead CLA
* Get some lawyer to create a CLA so Getahead can grant rights to the Foundation (or something similar)
* Get someone to pony up and let us move to SVN
* Unit tests

JSON support: One goal is a RESTian API so you can do something like this: http://example.com/dwr/json/ClassName/methodName?param1=fred;param2=jim and DWR will reply with a JSON structure containing the result of calling className.methodName("fred", "jim"); It would be good to support JSONP along with this. We might also allow POSTing of JSON structures, although I’m less convinced about this because it quickly gets DWR specific, and then what’s the point of a standard. Status - DWR has always used a superset of JSON that I like to call JavaScript. We do this to cope with recursive data, XML objects, and such like. I’ve done most of the work so that DWR can use the JSON subset, but not created the ‘handler’ to interface between the web and a JSON data structure.

Bayeux Support: Greg Wilkins (Jetty) committed some changes to DWR, which need some tweaks to get working properly. Greg still intends to complete this.

File/Image Upload and Download: This allows a Java method to return an AWT BufferedImage and have that image turn up in the page, or to take or return an InputStream and have that populated from a file upload or offered as a file download. I’ve had some bug reports that it doesn’t work with some browsers, also we need to find a way to report progress to a web page simply.

DWR Hub and integration with JMS and OpenAjax Hub: We have a hub, along with one way integration with JMS. The OpenAjax portion will be simple except for the getting the OpenAjax Hub to work smoothly with JMS part. Much of this work has not hit CVS yet, but will do soon.

Reverse Ajax Proxy API Generator: The goal with this is a program that will take JavaScript as input, and output a Java API which, when called, generates JavaScript to send to a browser. Some of this work has been tricky, but then meta-meta-programming was always bound to be hard. This currently mostly works with TIBCO GI, but more work will be needed to allow it to extract type information from other APIs.

DOM Manipulation Library: Currently this is limited to window.alert, mostly because I’m not sure how far to take it. There are a set of things like history, location, close, confirm that could be useful from a server, and that are not typically abstracted by libraries.

Gears Integration: I’ve not started this, but it needs to take higher priority than it currently does. It would be very cool if DWR would transparently detect Gears, and then allow some form of guaranteed delivery including resending of messages if the network disappears for a while.

Website: We need to get the DWR website moved away from the Getahead server, and onto Foundation servers. There will be some URLs to alter as part of this, and I don’t want to lose Google juice by doing it badly.
The documentation for DWR 2 was not up to the standards of 1.x, and while it has been getting better, we could still do more. One thing that has held this back has been lack of a DWR wiki. I hope we can fix this with the server move.

Source Repo: We are currently using CVS hosted by java.net (which is a collab.net instance - yuck). They support SVN, but want to charge me a few hundred dollars to upgrade. Maybe the Foundation can either ridicule them into submission or pay the few hundred dollars for the meta-data so we can host the repo. ourselves. The latter option is probably better.

Unit Tests: I've been trying for ages to find a way to automatically test with multiple browsers and servers. WebDriver looked good for a while, but it doesn't look like the project is going anywhere particularly quickly, so I'm back trying to get Selenium to act in a sane way.

PostgreSQL 8.3 vs. 8.2 - a simple benchmark

Tagged:  

I ran across an interesting post that benchmarks the newest version of PosrgreSQL (8.3) versus the previous release (8.2). PostgreSQL is one of the most used databases for web 2.0 startups (behind MySQL).

The below is from the post.

With 8.3 just around the corner more and more people are actually starting to test 8.3 with their code base and wondering if it will be worth to switch/upgrade and so did I.
The following is not really an in-depth benchmark but meant as a simple testing of 8.3 on a very specific (but not uncommon) workload and with small set of different configuration parameters.

All the testing is done on a DL380 G5 with two Quadcore Intel Xeon CPUs (X5345 - so 2,33Ghz per core) 12GB of RAM and 6 disks in a RAID1+0 with 512MB BBWC on a Smartarray P400.
The box is running Debian/Etch AMD/64 with the debian supplied kernel (2.6.18 more or less) and a current -HEAD snapshot of PostgreSQL (more or less BETA4 code) compiled as a 64bit binary.

The benchmark database is initialized with a scaling factor of 100 (equals to 10M rows) which seems to be a reasonable size for a table in an OLTP style database) all testing was done with 100 clients and 100000 transactions/client which comes out to 10M transactions and an average runtime of about 1,5 hours.
The first test is a simple comparison of 8.2 vs 8.3B4 in both the default (ie. unchanged postgresql.conf) and one with a somewhat reasonable tuning:

* checkpoint_segments=192
* maintenance_work_mem=128MB
* shared_buffers=1536MB
* wal_buffers=1024kB
* effective_cache_size=3084MB
* filesystem(ext3) mounted with noatime

82vs83b4.gif

what we can see here is that 8.3B4 is about 2.2x faster than 8.2 out-of-the-box for this very update heavy workload and is able to keep that advantage after similar tuning is applied to both instances.
The main reason for this rather large improvement is HOT. pgbench is one of the workloads that benefit most from it and a bit of preliminary testing with similiar real-life workloads (session tables or stuff like the SQL-Bayes support in Spamassassin) show equally impressive gains.

the second thing I tried to test was the effect of various shared_buffer settings on the transaction rate(the other parts of the configuration stayed the same as in the "tuned" variant above):

83b4shm.gif

there are two main things to note here - one is the obvious one that the scaling on the x-axis is not linear.
The other one is that the best performance seems to be around 1,5GB of shared memory which is about the size of the database on-disk in 8.3. Higher settings to not help and in fact cause a slight performance degeneration.

Read the full post here

From these tests it looks like 8.3 is going to very good release. With databases performance tuning is critical to having your application run its best.

As always if you have any questions about the above you can comment on this post or message me on Social Ajaxonomy. Also if you have an interesting site, library or story you can use the link on the side bar of the blog (or click here) to submit the item.

PHP Templating

Tagged:  

If you've done any developing with PHP (especially with open source PHP projects) you will have probably at least seen some templating. A commonly used template engine for PHP is Smarty. Smarty does a good job of helping your PHP applications adhere to a Model View Controller design.

As you probably know Model View Controller is a way of abstracting portions of an applications so that you can have multiple people working on their part of the project without stepping on other peoples toes. Which in any decent sized organization (or even in a startup) is very helpful to the development process. If you develop in PHP I recommend that you use templating, even in Ajax applications with PHP on the backend I recommend using templating, as it helps abstract the design from both the backend code and the data (which is exactly what we want).

Just to get you started with templating I've included a portion of a good tutorial on templating using Smarty below.

Basic Templating

At the most basic level, templating can be achieved by removing the common HTML code into separate files and including them into any php file:

<?php include("header.php"); ?>
some content and html goes here
<?php include("footer.php"); ?>

This approach is fine for most purposes and does help reusing common HTML (and/or php) code, for menu generation and setting of page footers and headers. Unfortunately, it does not adapt particularly well to all situations. In the event of the page dealing with form handling or session usage, life becomes slightly more complex.

The relatively simple form below shows how the code can become relatively messy and difficult to follow.

<?php
session_start();
require_once("header.php");

if(isset($_SESSION['username'])) {
   echo "Logged in as : " . $_SESSION['username'];
}
if(isset($_POST['id'])) {
   $results = database_query("SELECT * FROM blah WHERE id = ?", Array($_POST['id']));
   if(sizeof($results>0)) {
      echo "<p>The following matches were found:</p>";
      echo "<ul>\n";
      foreach($results as $item) {
         echo "<li>" . $item . "</li>\n";
      }
      echo "</ul>\n";
   }
   else {
      echo "<p>No matches found.</p>";
   }
}
?>
<p>Please enter an ID to search for</p>
<form method='post'>
<input type='text' name='id' value=''>
<input type='submit'>
</form>
<?php
require_once("footer.php");
?>

Fom looking at the above code, we have the following problems :

* We have to do some things before we can include the header page (e.g. session_start() or header() calls).
* Ideally we should only be outputting HTML at the last possible moment
* We have a mixture of PHP and HTML code, which makes the file bulky and difficult to edit
* It wouldn't be very easy to use the same underlying code to generate a page as a .pdf or other format
* Upon adding more logic, it's likely that it will become harder to follow the flow of execution
* The over complex nature of the page means it's highly likely that bugs will be introduced

We can help to solve the above problems by splitting the page into two distinct parts :

* Logic (dealing with form parameters and data retrieval)
* Display (HTML or other output).

Smarty has the answer...

There are numerous templating engines available for PHP, and some projects have the discipline to use native PHP embedded within a file for display of logic. I found Smarty's existing documentation and framework to make the migration far easier, as well as forcing me to resist the temptation to embed PHP based logic within the template.

Smarty is written in PHP, and installation is a matter of downloading a compressed file, extracting it somewhere appropriate (E.g. /usr/local/lib/php) and creating the necessary directory structure within your project. In order to use Smarty from within a PHP page, the following is required :

<?php
require_once("path/to/Smarty.class.php");
$smarty = new Smarty();
....
?>

Jumping straight to the refactored example, the way Smarty can be used, and the benefits of it should become clear.

First, the template file which is used by Smarty (this will normally live in a directory called 'templates'). This file contains HTML markup and some simple Smarty specific syntax. This file has a suffix of .tpl. An example one follows :

{include file='header.php'}
<p>Welcome to the application</p>

{if isset $logged_in_as}
    <p>{$logged_in_as}</p>
{/if}

{if sizeof($results > 0)}
    <ul>
    {foreach from=$results item=i}
       <li>{$i}</li>
    {/foreach}
    </ul>
{/if}
<p>Please enter an ID to search for</p>
<form method='post'>
  <input type='text' name='id' value=''>
  <input type='submit'>
</form>
{include file='footer.php'}

As you can see, Smarty provides constructs for looping and checking for the existence of variables. Smarty's template language is very similar to PHP in many respects.

In order to populate this template, the following php code could be used :

<?php
require_once("path/to/Smarty.class.php");
$smarty = new Smarty();
if(isset($_SESSION['username'])) {
   $smarty->assign("logged_in_as", "You are logged in as " . $_SESSION['username']);
}
if(isset($_POST['id'])) {
   $results = database_query("SELECT * FROM blah WHERE id = ?", Array($_POST['id']));
   $smarty->assign("results", $results);
}
$smarty->display("our_template.tpl");
?>

As should be apparent, the assign function is used to pass data to the template for display, and the 'display' function is used to select which template we wish to use. Because there is no output to the browser until 'display' is called, we can use any of the functions that output HTTP headers to the browser throughout nearly all of the file without getting the 'headers already sent' error.

In addition to the above you can find more useful resources below.

This is the tutorial that the above was taken from.
This resource is from the Smarty web site and is extremely useful.

As always if you have any questions on this post you can either leave a comment or contact me through Social Ajaxonomy.

XML versus JSON - What is Best for Your App?

Tagged:  

One of the biggest debates in Ajax development today is JSON versus XML. This is at the heart of the data end of Ajax since you usually receive JSON or XML from the server side (although these are not the only methods of receiving data). Below I will be listing pros and cons of both methods.

If you have been developing Ajax applications for any length of time you will more than likely be familiar with XML data. You also know that XML data is very powerful and that there are quite a few ways to deal with the data. One way to deal with XML data is to simply apply a XSLT style sheet to the data (I won't have time in this post to go over the inconsistent browser support of XSLT, but it is something to look into if you want to do this). This is useful if you just want to display the data. However, if you want to do something programmatically with the data (like in the instance of a web service) you will need to parse the data nodes that are returned to the XMLHTTPRequest object (this is done by going through the object tag by tag and getting the needed data). Of course there are quite a few good pre-written libraries that can make going through the XML data easier and I recommend using a good one (I won't go into depth as to what libraries I prefer here, but perhaps in a future post). One thing to note is that if you want to get XML data from another domain you will have to use a server side proxy as the browser will not allow this type of receiving data across domains.

JSON is designed to be a more programmatic way of dealing with data. JSON (JavaScript Object Notation) is designed to return data as JavaScript objects. In an Ajax application using JSON you would receive text through the XMHTTPRequest object (or by directly getting the data through the script tag which I will touch on later) and then pass that text through an eval statement or use DOM manipulation to pass it into a script tag (if you haven't already read my post on using JSON without using eval click here to read the post). The power of this is that you can use the data in JavaScript without any parsing of the text. The down side would be if you just wanted to display the data there is no easy way to do this with JSON. JSON is great for web services that are coming from different domains since if you load the data through a script tag then you can get the data without a domain constraint.

The type of data that you use for your application will depend on quite a few factors. If you are going to be using the data programmatically then in most cases JSON is the better data method to use. On the other hand if you just want to display the data returned I would recommend XML. Of course there may be other factors such as if you are using a web service, which could dictate the data method. If you are getting data from a different domain and JSON is available this may be the better choice. For Ruby on Rails developers, if you would prefer to use JSON and XML is all that is available the 2.0 release allows you to change XML into JSON. One of the biggest reasons that people use JSON is the size of the data. In most cases JSON uses a lot less data to send to your application (of course this may very depending on the data and how the XML is formed).

I would recommend that you take a good look at the application that you are building and decide based on the above which type of data you should deal with. There may be more factors than the above including corporate rules and developer experience, but the above should have given you a good idea as to when to use either data method.

If you would like to contact me regarding any of the above you can make me your friend on Social Ajaxonomy and send a message to me through the service (Click here to go to my profile on Social Ajaxonomy).

Syndicate content