Raspberry Pi Pico Tips and Tricks

Wednesday 31 July 2013

Bare bones structure for a dc.js and crossfilter page

The following post is a portion of the D3 Tips and Tricks book which is free to download. To use this post in context, consider it with the others in the blog or just download the the book as a pdf / epub or mobi .
----------------------------------------------------------
To learn some of the capabilities of dc.js and crossfilter we will start with a rudimentary template and build chart examples as we go.
The template we'll start with will load d3.js, crossfilter.js, dc.js jquery.js and bootstrap.js. We will be including bootstrap as it provides lots of nice capabilities for fine tuning layout and styling as laid out in the chapter on using bootstrap. Since bootstrap depends on jquery, we have to load that as well.
We'll also load cascading style sheets for bootstrap and dc.js.
The template will load a csv file with earthquake data sourced from New Zealand's Geonet site over a date range that covers a period of reasonable activity in July 2013.
In it's bare bones form we will present only a data table with some values from the csv file. When we begin to add charts, we will see this table adjust dynamically.
We'll move through the explanation of the code in a similar process to the other examples in the book. Where there are areas that we have covered before, I will gloss over some details on the understanding that you will have already seen them explained in other sections.
Here is the full code (an expanded version of which can be downloaded from here with all the associated*.js and *.css files);
<!DOCTYPE html>
<html lang='en'>
<head>
  <meta charset='utf-8'>

  <title>dc.js Experiment</title>

  <script src='js/d3.js' type='text/javascript'></script>
  <script src='js/crossfilter.js' type='text/javascript'></script>
  <script src='js/dc.js' type='text/javascript'></script>
  <script src='js/jquery-1.9.1.min.js' type='text/javascript'></script>
  <script src='js/bootstrap.min.js' type='text/javascript'></script>

  <link href='css/bootstrap.min.css' rel='stylesheet' type='text/css'>
  <link href='css/dc.css' rel='stylesheet' type='text/css'>

  <style type="text/css"></style>
</head>

<body>

<div class='container' style='font: 12px sans-serif;'>
  <div class='row'>
    <div class='span12'>
      <table class='table table-hover' id='dc-table-graph'>
        <thead>
          <tr class='header'>
            <th>DTG</th>
            <th>Lat</th>
            <th>Long</th>
            <th>Depth</th>
            <th>Magnitude</th>
            <th>Google Map</th>
            <th>OSM Map</th>
          </tr>
        </thead>
      </table>
    </div>
  </div>
</div>
  
<script>
  
// load data from a csv file
d3.csv("data/quakes.csv", function (data) {

  // format our data
  var dtgFormat = d3.time.format("%Y-%m-%dT%H:%M:%S");
  
  data.forEach(function(d) { 
    d.dtg   = dtgFormat.parse(d.origintime.substr(0,19)); 
    d.lat   = +d.latitude;
    d.long  = +d.longitude;
    d.mag   = d3.round(+d.magnitude,1);
    d.depth = d3.round(+d.depth,0);
  });

  // Create the dc.js chart objects & link to div
  var dataTable = dc.dataTable("#dc-table-graph");

  // Run the data through crossfilter and load our 'facts'
  var facts = crossfilter(data);

  // Create dataTable dimension
  var timeDimension = facts.dimension(function (d) {
    return d.dtg;
  });

  // Setup the charts
  
  // Table of earthquake data
  dataTable.width(960).height(800)
    .dimension(timeDimension)
    .group(function(d) { return "Earthquake Table"
     })
    .size(10)
    .columns([
      function(d) { return d.dtg; },
      function(d) { return d.lat; },
      function(d) { return d.long; },
      function(d) { return d.depth; },
      function(d) { return d.mag; },
      function(d) { return '<a href=\"http://maps.google.com/maps?z=12&t=m&q=loc:' + d.lat + '+' + d.long +"\" target=\"_blank\">Google Map</a>"},
      function(d) { return '<a href=\"http://www.openstreetmap.org/?mlat=' + d.lat + '&mlon=' + d.long +'&zoom=12'+ "\" target=\"_blank\"> OSM Map</a>"}
    ])
    .sortBy(function(d){ return d.dtg; })
    .order(d3.ascending);

  // Render the Charts
  dc.renderAll();
  
});
  
</script>
    
</body>
</html>
The first part of the code starts the html file and inside the <head> segment loads our JavaScript and css files
<!DOCTYPE html>
<html lang='en'>
<head>
  <meta charset='utf-8'>

  <title>dc.js Experiment</title>

  <script src='js/d3.js' type='text/javascript'></script>
  <script src='js/crossfilter.js' type='text/javascript'></script>
  <script src='js/dc.js' type='text/javascript'></script>
  <script src='js/jquery-1.9.1.min.js' type='text/javascript'></script>
  <script src='js/bootstrap.min.js' type='text/javascript'></script>

  <link href='css/bootstrap.min.css' rel='stylesheet' type='text/css'>
  <link href='css/dc.css' rel='stylesheet' type='text/css'>

  <style type="text/css"></style>
</head>
 It's important to note here that the order of loading the files is important. The jquery-1.9.1.min.js file must be loaded before the bootstrap.min.js file or it just won't work.
From here we move into the section where we set up our page to load our bootstrap grid layout for the table.
<div class='container' style='font: 12px sans-serif;'>
  <div class='row'>
    <div class='span12'>
      <table class='table table-hover' id='dc-table-graph'>
        <thead>
          <tr class='header'>
            <th>DTG</th>
            <th>Lat</th>
            <th>Long</th>
            <th>Depth</th>
            <th>Magnitude</th>
            <th>Google Map</th>
            <th>OSM Map</th>
          </tr>
        </thead>
      </table>
    </div>
  </div>
</div>
It might look a little complicated, but if you have a look through the bootstrap chapter (where we cover using the bootstrap grid layout), you will find it no problem at all.
The important features to note are that we have declared an ID selector for our table id='dc-table-graph'and we have set a series of headers for the table; DTG, Lat, Long, Depth, Magnitude, Google Map and OSM Map.
We have also included some bootstrap styling for the table by including the the class='table table-hover' portion of the code. With that styling included our table looks like this;
Without the styling it would look like this;
We will be adding to this grid layout section as we add in charts which will want their own allocated space on our page.
The next section of the file starts our JavaScript and d3.js.
// load data from a csv file
d3.csv("data/quakes.csv", function (data) {

  // format our data
  var dtgFormat = d3.time.format("%Y-%m-%dT%H:%M:%S");
  
  data.forEach(function(d) { 
    d.dtg   = dtgFormat.parse(d.origintime.substr(0,19)); 
    d.lat   = +d.latitude;
    d.long  = +d.longitude;
    d.mag   = d3.round(+d.magnitude,1);
    d.depth = d3.round(+d.depth,0);
  });
We load our csv file with the line d3.csv("data/quakes.csv", function (data) {. I have deliberately left this file in its raw form as received from Geonet. Its format looks a little like this;

FID,publicid,origintime,longitude,latitude,depth,magnitude,magnitudetype,status,phases,type,agency,updatetime,origin_geom
quake.2013p550753,2013p550753,2013-07-23T18:41:11.707,174.4298,-41.5313,7.9883,2.2425,M,automatic,27,,WEL(GNS_Primary),2013-07-23T18:43:15.672,POINT (174.42978 -41.531299)
quake.2013p550747,2013p550747,2013-07-23T18:38:02.481,174.414,-41.5181,11.6797,1.7892,M,automatic,11,,WEL(GNS_Primary),2013-07-23T18:39:25.37,POINT (174.41398 -41.518114)
quake.2013p550725,2013p550725,2013-07-23T18:26:30.229,175.5516,-40.0264,8.75,3.4562,M,automatic,21,,WEL(GNS_Primary),2013-07-23T18:29:46.305,POINT (175.55155 -40.026412)
We then declare a small function that will format our time correctly (var dtgFormat = d3.time.format("%Y-%m-%dT%H:%M:%S");). This follows exactly the same procedure we took when creating our very first simple line graph at the start of the book.
T> However, there is a slight twist... Observant readers will notice that while we have a function that resolves a date/time that is formatted with year, month, day, hour, minute and second values, I don't include an allowance for the fractions of seconds that appear in the csv file. Well spotted. The reason for this is that in spite of initially including this formatting, I found it caused some behaviour that I couldn't explain, so I reverted to cheating and you will note that in the next section when I format the values from the csv file, I truncate the date/time value to the first 19 characters (d.origintime.substr(0,19)). This solved my problem by chopping off the fractions of a second (admittedly without actually solving the underlying issue) and I moved on with my life.
As mentioned. the next section goes through each of the records and formats them correctly. The date/time gets formatted, the latitude and longitude are declared as numerical values (if they weren't already) and the magnitude and depth valued are rounded to make the process of grouping them simpler.
  data.forEach(function(d) { 
    d.dtg   = dtgFormat.parse(d.origintime.substr(0,19)); 
    d.lat   = +d.latitude;
    d.long  = +d.longitude;
    d.mag   = d3.round(+d.magnitude,1);
    d.depth = d3.round(+d.depth,0);
  });
The next section in our code sets up the dimensions and groupings for the dc.js chart type and crossfilter functions
  // Create the dc.js chart objects & link to div
  var dataTable = dc.dataTable("#dc-table-graph");

  // Run the data through crossfilter and load our 'facts'
  var facts = crossfilter(data);

  // Create dataTable dimension
  var timeDimension = facts.dimension(function (d) {
    return d.dtg;
  });
The first line assigns the variable dataTable to the dc.js dataTable chart type (var dataTable = dc.dataTable("#dc-table-graph");) and assigns the chart to the ID selectordc-table-graph.
Then we load all of our data into crossfilter (var facts = crossfilter(data);) and give it the name facts.
Lastly we create a dimension from our data (facts) of the date/time values.


  var timeDimension = facts.dimension(function (d) {
    return d.dtg;
  });
The last major chunk of code is the piece that configures our data table.
  dataTable.width(960).height(800)
    .dimension(timeDimension)
    .group(function(d) { return "Earthquake Table"
     })
    .size(10)
    .columns([
      function(d) { return d.dtg; },
      function(d) { return d.lat; },
      function(d) { return d.long; },
      function(d) { return d.depth; },
      function(d) { return d.mag; },
      function(d) { return '<a href=\"http://maps.google.com/maps?z=12&t=m&q=loc:' + d.lat + '+' + d.long +"\" target=\"_blank\">Google Map</a>"},
      function(d) { return '<a href=\"http://www.openstreetmap.org/?mlat=' + d.lat + '&mlon=' + d.long +'&zoom=12'+ "\" target=\"_blank\"> OSM Map</a>"}
    ])
    .sortBy(function(d){ return d.dtg; })
    .order(d3.ascending);
Firstly the width and height are declared (dataTable.width(960).height(800)). Then the dimension of the data that will be used is declared (.dimension(timeDimension)).
Separate sections of the table can have a header applied. In this case the entire table is given the grouping 'Earthquake Table' (.group(function(d) { return "Earthquake Table"})), but several examples online give the date.
The .size(10) line sets the maximum number of lines of the table to be displayed to 10.
Then we have the block of code that sets what data appears in which columns. It should be noted that this matches up with the headers that were declared in the earlier section of the code where the div's for the table were laid out.
The portion of this block that has a '*little bit of fancy*' are the two columns that set links that allow a user to click on the designation 'Google Map' or 'OSM Map' and have the browser open a new window containing a Google or Open Street Map (OSM) map with a marker designating the location of the quake. I won't mention too much about how the links are made up other than to say that they are pretty much a combination of the latitude, longitude and zoom level for both. Please check out the code for more.
Lastly we sort by the date/time value (.sortBy(function(d){ return d.dtg; })) in ascending order (.order(d3.ascending);).
The final part of our JavaScript renders all our charts (dc.renderAll();) and then closes off the initial d3.csv call.
  // Render the Charts
  dc.renderAll();
  
});
The final part of our code simply closes off the <script><body> and <html> tags.
There we have it. The template for starting to play with different crossfiltered dc.js charts.
The description above (and heaps of other stuff) is in the D3 Tips and Tricks book that can be downloaded for free (or donate if you really want to :-)).

Tuesday 30 July 2013

Introduction to dc.js

The following post is a portion of the D3 Tips and Tricks book which is free to download. To use this post in context, consider it with the others in the blog or just download the the book as a pdf / epub or mobi .
----------------------------------------------------------
Why, if we've just explored the benefits of crossfilter are we now introducing a completely different JavaScript library (dc.js)?
Well, crossfilter isn't a library that's designed to draw graphs. It's designed to manipulate data. D3.js is a library that's designed to manipulate graphical objects (and more) on a web page. The two of them will work really well together, but the barrier to getting data onto a web page can be slightly daunting because the combination of two non-trivial technologies can be difficult to achieve.
This is where dc.js comes in. It was developed by Nick Qi Zhu and the first version was released on the 7th of July 2012.
Dc.js is designed to be an enabler for both libraries. Taking the power of crossfilter's data manipulation capabilities and integrating the graphical capabilities of d3.js.
It is designed to provide access to a range of different chart types in a relatively easy to use fashion. It is more limited in the range of options available for graphical design in this respect than d3.js, but the simplicity that it provides for creating pages using crossfiltered data is a real benefit if you're anything like me and need all the help you can get.
The different (generic) types of chart that dc.js supports are
  • Bar Chart
  • Pie Chart
  • Row Chart
  • Line Chart
  • Bubble Chart
  • Geo Choropleth Chart
  • Data Table
All these examples come with a range of options which we will cover in greater depth in later sections.
My initial sources of information for developing the examples here came primarily from;

Bar Chart

This is a standard bar chart.

Pie Chart

This is a standard pie chart. The examples below are from one of Nick Zhu's dc.js example pages.

Row Chart

The row chart is a horizontal version of a bar chart, but with the ability to represent discrete values and to be able to select them for filtering by clicking on them. The example below is from one of Nick Zhu's dc.js example pages

Line Chart

Standard line chart.

Bubble Chart

The bubble chart is a derivative of a scatter plot with control over x axis position, y axis position, bubble radius and colour.

Geo Choropleth Chart

A Choropleth map is one where areas are shaded or patterned in proportion to the measurement of a variable being displayed on the map, such as population density or per-capita income. The example below is from one of Nick Zhu's dc.js example pages

Data Table

A data table is a simple table made up of data elements derived from the information loaded.

The description above (and heaps of other stuff) is in the D3 Tips and Tricks book that can be downloaded for free (or donate if you really want to :-)).