Caching a Back-end Data Store on the Front-end with Local Storage

I’m working on a small web application as a side project at the moment, the app works with a large-ish dataset that will only change every now and then.  This problem is begging to be fixed with HTML5 local storage, so I’ve put together a POC to see how it can work before bringing it into the project.

To do this, I’m going to use MySQL as the data source with a quick and dirty PHP script acting as a controller and model hybrid, and I’ll use jQuery to handle the AJAX.  But this solution can be used across just about any web stack, whether you use a relational database,  or any database at all.  I’m providing an overview of the method, not a detailed guide on a rigid implementation to be followed to the T.

Firstly, I should define what I mean by cache

  • Data will be initially sent to the browser
  • When the user returns to the app, it checks if the data has changed
  • If the data hasn’t changed, the app doesn’t cycle the data
  • If the data has changed, cycle it (trash the cache, download the data fresh from the server)

So we need to tell if the data has changed between the cache and the data source.  We’ll use a hash for that.

I’m using this very simple table in the example:

some_table

Column Type
id int(11)
some_data varchar(10)

I use the following SQL to get a hash of the current state of this table:

SELECT SHA1(CONCAT(GROUP_CONCAT('id', 'some_data'))) AS hash FROM some_table

SHA1 is the hash algorithm – it doesn’t matter what algorithm we use here
CONCAT will flatten the values we get back into one long string ready for the hashing algorithm
CONCAT_GROUP will flatten the results to a single row

From this, we’ll get a hash like this:

0fef95f4087ff39fab1bf51915c9a2650995de5d

Then when we do this:

INSERT INTO some_tables (some_data) VALUES ('this is some data')

And re-run the SELECT query, we get:

9a2accaac936a97f39822232fad6e3d7907273e9

And so on for any other changes to the database that result in a new instance or state of the table.

Below is the Javascript/jQuery we use when the page loads:

$(document).ready(function()
{
$("#data-source").html("");

$.ajax({
type : 'GET',
url : 'get_data.php',
data : {
'ajax' : true,
'hash' : window.localStorage.getItem('hash')
}
}).done(function(response) {
var data = '';

if(typeof(Storage)!=="undefined")
{
/**
* If we recieve back the same hash we sent, the data hasn't updated
* and has not been sent in the response
*/
if(response.hash != window.localStorage.getItem('hash'))
{
if(window.localStorage.getItem('hash')===null)
{
console.log("First time data+hash received by server");
}
else
{
console.log("Updating local storage from server");
}

window.localStorage.setItem('hash', response.hash);
window.localStorage.setItem('data', response.data);
}
else
{
console.log("No updates, using local storage")
}

/**
* This has either just been updated, or is the same as last time
*/
data = window.localStorage.getItem('data');
}
else
{
console.log("Updated from server, no local storage");

/**
* No cache, just load the data, which will have been sent to us
* because the hash is null.
*/
data = response.data;
}

console.log(data);
});
});

The referenced get_data.php will check the hash sent back from the local storage, if it’s null, it will send through a hash and the data. If the hash sent from local storage matches the current hash for the database table, it will only send back the hash (which will match what was sent from local storage).  And if the hashes do not match, it will send back a new hash and the data.

Here’s get_data.php:


if(isset($_GET['ajax']))
{
$responseObj = new stdClass();

if(!empty($_GET['hash']))
{
$current_hash = get_current_hash();

if($_GET['hash']==$current_hash)
{
/**
* The data is the same, so just send back the current hash for the
* front end to use for verification of no change
*/
$responseObj->hash = get_current_hash();
}
else
{
/**
* The data has changed, send back the new hash for the new data
* and of course, the new data
*/
$responseObj->hash = get_current_hash();
$responseObj->data = get_data();
}
}
else
{
/**
* There's no hash, must be a first time load or reload, send everything
*/
$responseObj->hash = get_current_hash();
$responseObj->data = get_data();
}

header('Content-type: application/json');
print json_encode($responseObj);
}
else
{
print get_data();
}

function get_data()
{
return array('key' => 'value');
}

function get_current_hash()
{
$link = mysqli_connect('localhost', 'user', 'pass');

mysqli_select_db($link, 'localcache');

$get_hash_sql = get_hash_sql();

$result = mysqli_query($link, $get_hash_sql);

$row = mysqli_fetch_assoc($result);

return $row['hash'];
}

function get_hash_sql()
{
return "SELECT SHA1(
CONCAT(GROUP_CONCAT('id', 'some_data'))) AS hash FROM some_table";
}

Here’s an overview:

Screen Shot 2013-11-11 at 9.09.05 PM

 

Open Data, Open Governence

Tonight I swung by my old uni for a talk on Open Data and Open Governance by Pia Waugh.

It was interesting to hear what is being done by the Australian government to make open the data it collects and produces. I generally got the feeling we’re at least heading in the right direction, and generally that feeling was given by words like de-identified public data being coupled with words like API.

Other interesting things from the talk:

A paragraph from the Coalition’s plan for the digital economy and e-Government:

Seek to ensure every Government interaction that occurs more than 50,000 times
per year can be achieved online by 2017. Video-conferencing via technologies
such as WRTC will be an acceptable substitute for physical proximity in most
cases.

 

This website: http://www.openaustralia.org/

 

The fact it can be easier to make information public than to share between government departments.  And that can be an all round good thing.

 

This hackathon that you probably already know about: GovHack

 

And that http://data.gov.au/ will be taking requests for data sets as of next week.

 

 

 

 

Move custom colors into a Category on UIColor

I’ve been working on a project with a large amount of custom colors defined programatically for the UI, meaning a large amount of my View Controller code started to look like this…

  1. thing.backgroundColor = [UIColor colorWithRed:39.0/255.0
  2.     green:174.0/255.0
  3.     blue:96.0/255.0
  4.     alpha:1.0];

Repeat it n times, and your methods are getting bloated with RGBa like above.

To clean things up a bit, I create a Category on UIColor and defined all my custom colors in methods on that category.

  1. @interface UIColor (ProjectColors)
  2. + (UIColor *)projectGreen;
  3. + (UIColor *)projectBlue;
  4. + (UIColor *)twitterBlue;
  5. + (UIColor *)facebookBlue;
  6. @end
  1. @implementation UIColor (ProjectColors)
  2. + (UIColor *)projectGreen
  3. {
  4. return [UIColor colorWithRed:39.0f/255.0f
  5.     green:174.0f/255.0f
  6.     blue:96.0f/255.0f
  7.     alpha:1.0f];
  8. }
  9. @end

Now my View Controller isn’t full of RGBa, and those custom colors now have nice names I can refer to later.

When I want the green I use for this project, I get it like this:

  1. thing.backgroundColor = [UIColor projectGreen];

Logging front-end errors with your usual CodeIgniter logging.

If you use CodeIgniter for your PHP apps, you probably use its nice logging system too, i.e. log_message(”, ”). After having some strange front end problems with a site I built recently, I wanted to get logs of errors happening on the front-end, and why not log them right in with the log files I monitor for backend issues anyway?

Put this somewhere in your front-end:

  • <script>
  • window.onerror = function(errorMsg, url, lineNumber)
  • {
  •     var log = new Image();
  •     var baseURL = ‘yourURLHere’ + ‘/felog’;
  •     log.src = baseURL + ‘?msg=’ + errorMsg + ‘&url=’ + url + ‘&ln=’ + lineNumber;
  • };
  • </script>

I adapted the above snippet from this article: http://devblog.xing.com/frontend/how-to-log-javascript-errors/

Then put this in your back end (make sure baseURL in the js above matches up to this)

  • <?php if ( ! defined(‘BASEPATH’)) exit(‘No direct script access allowed’);
  • class Felog extends CI_Controller {
  •     public function log()
  •     {
  •         header(“Content-Type: image/jpg”);
  •         $error_message = $this->input->get(‘msg’);
  •         $error_url = $this->input->get(‘url’);
  •         $error_line = $this->input->get(‘ln’);
  •         log_message(‘error’, ‘Front end error:’);
  •         log_message(‘error’, ‘ – ‘ . $error_message);
  •         log_message(‘error’, ‘ – At: ‘ . $error_url);
  •         log_message(‘error’, ‘ – Line: ‘ . $error_line);
  •         log_message(‘error’, ‘ – UA String: ‘ . $_SERVER['HTTP_USER_AGENT']);
  •     }
  • }
  • /* End of file felog.php */
  • /* Location: ./application/controllers/felog.php */

Boom. Front-end errors in your CodeIgniter logs.

Converting between doc/docx/pdf/html/etc en masse

Too useful not to write a quick post for the next person Googling and messing around in the shell trying to get this done quicker than manually opening and exporting the files by hand.

unoconv (Github here) is a Python script that converts documents between one format and another, including doc/docx to html, pdf to html, etc.  unoconv works by spawning instances of LibreOffice, so if you don’t have it already get that from http://www.libreoffice.org/ there, install, and then run up:

  1. find ./ -name “*.docx” -exec python unoconv -f html ‘{}’ \;

The above assumes you’re running from the same directory as the unoconv script and the documents you want to convert. It also does the conversion to matching files in all subdirectories. Change the paths above to suit your setup.

…oh, it also assumes you’re running *nix ;)  I did this on MacOS.

 

 

 

2D Collision Detection Using UIViews, animateWithDuration: And CA Layers

Sometimes, especially times when you’re trying to create a basic iOS game using standard UIKit and no real engine, detecting collisions between UIViews animated using animateWithDuration is very handy.

I’ve just uploaded a very basic (and somewhat unrealistic) example of how to do this to my Github: https://github.com/tobygundry/UIView2DCollisionDetection

There’s a lot of boilerplate in there, so I won’t turn this post into a complete tutorial style one.  The important parts are… Continue reading

New App Waiting for Review

I put together a simple ping utility app for testing 2 components of a larger app I’ve been working on. Once it was made, I decided it would be a cool free app to put on the App Store, did some tidy up of the code, made the UI pretty, and got it ready for submission.  The app is called ‘ping-a-ling’, and does nothing more but ping the host you enter until you tell it to stop (or it encounters an error.)

Screen Shot 2013-07-02 at 12.07.26 PM

It’s a universal binary that runs on both iPhone and iPad.  I have no plans for it other than to keep it stable.  Hopefully some people will find it useful and appreciate the totes awesome flat UI.  I’ll update this page when it moves from ‘Waiting for Review’ to ‘Ready for Sale’.

Maintain in-context editing for javascript heavy Concrete5 themes

If your Concrete5 theme uses a lot of Javascript or AJAX page loading or anything that gets in the way of Concrete’s in-context editor, just temporarily remove the Javascript…

Assuming the effects in your Concrete5 theme degrade gracefully to a fully usable, crawlable, legible website when javascript isn’t available (and it does, right?) then simply switching off your javascript when the user is in ‘edit mode’ will place the content in a static way that the user can access and edit as Concrete5 intended.

Just put your javascript between this code…

  1. <?php
  2. global $cp;
  3. if (!$cp->canViewToolbar()) {
  4. ?>
  5. <script src="the.js"></script><!-- your javascript -->
  6. <?php>
  7. }
  8. ?>

When your admin users are logged in, the javascript that’s stopping them from seeing what they want to edit, just won’t be there. It’s a simple method that doesn’t affect your markup or your javascript or see you adding unnecessary logic all over your theme files.