Raspberry Pi Pico Tips and Tricks

Friday 25 April 2014

Adding in zero values into a time series in d3.js

The following post is a portion of the D3 Tips and Tricks book which is free to download. To use this post in context, consider it with the others in the blog or just download the the book as a pdf / epub or mobi .
----------------------------------------------------------

Padding for zero values

The ‘Padding for zero values’ question is one that I posted on Stack Overflow in April 2014 and which was elegantly answered by the user 'explunit'.
The premise is that I want to be able to graph points in a time series that has regular intervals but where some of those intervals are missing. For example a line graph of the following data (notice that August 2013 is missing)…
date,value
2013-01,53
2013-02,165
2013-03,269
2013-04,344
2013-05,376
2013-06,410
2013-07,421
2013-09,376
2013-10,359
2013-11,392
2013-12,433
2014-01,455
2014-02,478
… would produce a graph that looked like the following
Interpolated time series
But if the reason for the missing August month was that the value would have been zero, then the graph should actually look like this;
What actually happened
Now, this may sound like a contrived example, but I have come across it in cases where querying for data from a MySQL database by counting (COUNT(*)) the number of instances of an event in a time series (number of sales of an individual item on a monthly basis for instance). The end result is that you are left with a list of months with values for those months where sales occurred, but where no sales occurred, there is no result at all. This can be overcome by some serious MySQL-fu or PHP trickery, but every solution I implemented had a ‘cracking a walnut with a sledgehammer’ feel about it.
Now I know that shifting the resolution of this problem from the server to the client might not sit well for everyone, but that doesn’t mean it’s not a suitable solution.
The solution that was provided by the user ‘explunit’ has a nice natural feel to it and in spite of introducing an additional JavaScript library to load (Lo-Dash) it’s a solution that I will be using in the future for this type of problem. You will need to bear in mind that the lodash.js library will need to be loaded to enable this solution.
<script src=
  "http://cdnjs.cloudflare.com/ajax/libs/lodash.js/2.4.1/lodash.min.js">
</script>

Lo-Dash

Lo-Dash is a utility JavaScript library released under the MIT license that provides a range of useful functional programming helpers such as low level functions that can be used for iteration support for arrays, strings, objects, and arguments objects. It is often compared to underscore.js in terms of functionality.
_.FIND
_.find(collection, [callback=identity], [thisArg])
The ` _.find` function iterates over elements of a collection, returning the first element that the callback returns ‘truey’ for. The callback is bound to thisArg and invoked with three arguments; (value, index|key, collection).
If a property name is provided for callback the created “_.pluck” style callback will return the property value of the given element.
If an object is provided for callback the created “_.where” style callback will return true for elements that have the properties of the given object, else false.
Arguments
  1. collection (Array|Object|string): The collection to iterate over.
  2. [callback=identity] (Function|Object|string): The function called per iteration. If a property name or object is provided it will be used to create a “.pluck” or “.where” style callback, respectively.
  3. [thisArg] (*): The this binding of callback.

The explunit method

The following is the layout of the critical part of the code that the explunit method uses.
// Scale the range of the data
x.domain(d3.extent(data, function(d) { return d.date; }))
     .ticks(d3.time.month);
y.domain([0, d3.max(data, function(d) { return d.value; })]);

// Populate the new array
var newData = x.ticks()
               .map(function(monthBucket) { 
                   return _.find(data, 
                       { date: monthBucket }) || 
                       { date: monthBucket, value: 0 };
                });
It can be thought of as breaking the problem down into two steps. Firstly a new array is built that covers the range of values declared in the data and contains dates at a specified interval. Secondly, the script iterates over the old data and where the date matches a date in the new array it maps the value from the old array to the new array. Where there is no match, it maps the value of 0.
BUILD THE ARRAY
scale.ticks([count])
The domain in the x axis is set using the following line;
x.domain(d3.extent(data, function(d) { return d.date; }))
     .ticks(d3.time.month);
Here the extent of the domain is declared in a standard way (d3.extent(data, function(d) { return d.date; })) but by including the .ticks function the domain is broken into uniformly spaced time intervals defined by the specified time interval (in this case months per the d3.time.month statement). This has just created an array across the x axis with all the month values!
POPULATE THE ARRAY
In the next section we generate our new array (newData) with the values from the original data array and where necessary (there’s no value0 in the value field.
It’s all contained within the following code;
var newData = x.ticks()
               .map(function(monthBucket) { 
                   return _.find(data, 
                       { date: monthBucket }) || 
                       { date: monthBucket, value: 0 };
                });
x.ticks()
The x.ticks() portion recalls our new padded array of months.
.map([object])
In a wider (JavaScript) sense the map() method creates a new array by copying all enumerable properties from the object into the new array. This is implemented with .map which is used here;
               .map(function(monthBucket) { 
                   return _.find(data, 
                       { date: monthBucket }) || 
                       { date: monthBucket, value: 0 };
                });
In the example we are examining, the [object] is provided by the passing the date from the x.ticks() function as monthBucket. Whatever is returned by the function will get placed into newData. It’s within this function that we use the Lo-Dash _.find utility.
In common speak _.find will look over over the objects in an array (data) and will return the the first instance that it finds which matches its specified patten (in our case it will match whenever the date element in dataequals an array element specified by our x.ticks() (monthBucket) values or (if no match is made) it will return the date element from monthBucket and the value of 0.
The end result is an array called newData of objects containing equally spaced date elements across the range that we provided at an interval that we specify with values corresponding to our original array data where it matches the date in our new array and where it doesn’t, we introduce the value 0.

The description above (and heaps of other stuff) is in the D3 Tips and Tricks book that can be downloaded for free (or donate if you really want to :-)).

Saturday 19 April 2014

Using HTML inputs with d3.js

The following post is a portion of the D3 Tips and Tricks book which is free to download. To use this post in context, consider it with the others in the blog or just download the the book as a pdf / epub or mobi .
----------------------------------------------------------

Using HTML inputs with d3.js

Part of the attraction of using technologies like d3.js is that it expands the scope of what is possible in a web page. At the same time, there are many different options for displaying content on a page and plenty of ways of interacting with it.
Some of the most basic of capabilities has been the use of HTML entities that allow the entry of data on a page. This can take a range of different forms (pun intended) and the <input> tag is one of the most basic.

What is an HTML input?

An HTML input is an element in HTML that allows a web page to input data. There are a range of different input types (with varying degrees of compatibility with browsers) and they are typically utilised inside a <form>element.
For example the following code allows a web page to place two fields on a web page so that a user can enter their first and last names in separate boxes;
<form>
  First name: <input type="text" name="firstname"><br>
  Last name: <input type="text" name="lastname">
</form>
The page would then display the following;
A form input
The range of input types is large and includes;
  • text: A simple text field that a user can enter information into.
  • radio: Buttons that let a user select only one of a limited number of choices.
  • button: A clickable button that can activate JavaScript.
  • range: A slider control for setting a number whose exact value is not important.
  • number: A field for entering a number or toggling a number up and down.
… and many more. To check out others and get further background, it would be worth while visiting the Mozilla developer pages or w3schools.com.
While d3.js has the power to control and manipulate a web page to an extreme extent, sometimes it’s desirable to use a simple process to get a result. The following explanations will demonstrate a simple use case linking an HTML input with a d3.js element and will go on to provide examples of using multiple inputs, affecting multiple elements and using different input types. The examples are deliberately kept simple. They are intended to demonstrate functionality and to provide a starting position for you to go forward :-).

Using a range input with d3.js

The first example we will follow will use a range input to adjust the radius of a circle.
Adjust the radius of a circle

THE CODE
The following is the full code for the example. A live version is available online at bl.ocks.org or GitHub. It is also available as the file ‘input-radius.html’ as a separate download with D3 Tips and Tricks. A a copy of most the files that appear in the book can be downloaded (in a zip file) when you download the book from Leanpub.
<!DOCTYPE html>
<meta charset="utf-8">
<title>Input test (circle)</title>
  
<p>
  <label for="nRadius" 
         style="display: inline-block; width: 240px; text-align: right">
         radius = <span id="nRadius-value"></span>
  </label>
  <input type="range" min="1" max="150" id="nRadius">
</p>

<script src="http://d3js.org/d3.v3.min.js"></script>
<script>

var width = 600;
var height = 300;
 
var holder = d3.select("body")
      .append("svg")
      .attr("width", width)    
      .attr("height", height); 

// draw the circle
holder.append("circle")
  .attr("cx", 300)
  .attr("cy", 150) 
  .style("fill", "none")   
  .style("stroke", "blue") 
  .attr("r", 120);

// when the input range changes update the circle 
d3.select("#nRadius").on("input", function() {
  update(+this.value);
});

// Initial starting radius of the circle 
update(120);

// update the elements
function update(nRadius) {

  // adjust the text on the range slider
  d3.select("#nRadius-value").text(nRadius);
  d3.select("#nRadius").property("value", nRadius);

  // update the circle radius
  holder.selectAll("circle") 
    .attr("r", nRadius);
}

</script>
THE EXPLANATION
As with the other examples in the book I will not go over some of the simpler lines of code that are covered in greater detail in earlier sections of the book and will concentrate on those sections that contain new concepts, code or look like they might need expanding :-).
The first section is the portion that sets out the html range input;
<p>
  <label for="nRadius" 
         style="display: inline-block; width: 240px; text-align: right">
         radius = <span id="nRadius-value"></span>
  </label>
  <input type="range" min="1" max="150" id="nRadius">
</p>
The entire block is enclosed in a paragraph (<p>) tag so that is appears on a single line. It can be broken down into the label that occurs before the input slider which is given the id nRadius-value and the input proper.
The for attribute of the label tag equals to the id attribute of the input element to bind them together. This allows us to update the text later as the slider is moved.
The input tag can include four attributes that specify restrictions on the operation of the slider;
  • max: specifies the maximum value allowed
  • min: specifies the minimum value allowed
  • step: specifies the number intervals as you move the slider
  • value: Specifies the default value
The ids supplied for both the label and the input are important since they provide the reference for our d3.js script.
The first portion of our JavaScript is fairly routine if you’ve been following along with the rest of the book.
var width = 600;
var height = 300;
 
var holder = d3.select("body")
      .append("svg")
      .attr("width", width)    
      .attr("height", height); 

// draw the circle
holder.append("circle")
  .attr("cx", 300)
  .attr("cy", 150) 
  .style("fill", "none")   
  .style("stroke", "blue") 
  .attr("r", 120);
We append an SVG element to the body of our page and then we append a circle with some particular styling to the SVG element.
Then things start to get more interesting…
d3.select("#nRadius").on("input", function() {
  update(+this.value);
});
We select our input using the id that we had declared earlier in the html (nRadius). Then we use the .onoperator which adds what is called an ‘event listener’ to the element so that when there is a change in the element (in this case an adjustment of the slider of the input) a function is called (function()) that in turn calls the update function with the value from the input (+this.value). We haven’t seen the update function yet, but never fear, it’s coming.
We also call the update function with a specific value in the next line;
update(120);
This might seem slightly redundant, but unless the function gets a value, the text associated with the range input doesn’t get a reading and remains on ‘…’ until the slider is moved.
Lastly we have our update function;
function update(nRadius) {

  // adjust the text on the range slider
  d3.select("#nRadius-value").text(nRadius);
  d3.select("#nRadius").property("value", nRadius);

  // update the circle radius
  holder.selectAll("circle") 
    .attr("r", nRadius);
}
The first part of the function selects the label associated with our input (with the idnRadius-value) and applies the vaule that has been passed into the function (nRadius). The next line selects the input itself and applies the value to it (this would be the equivalent of having value="<number here>" as a property in the html).
Lastly, we select the circle element and apply the new radius value based on our input value nRadius(.attr("r", nRadius)).
And there we have it, a fully adjustable radius for our circle controlled with an HTML input.
Maximum radius for our circle

Using more than one input

In this example we will use two separate inputs (range type) to adjust the height and width of a rectangle.
Dual Inputs
This is not too much of a stretch from the previous single input example with the radius of a circle, but it may be useful to reinforce the concept and illustrate something slightly different.
THE CODE
The following is the full code for the example. A live version is available online at bl.ocks.org or GitHub. It is also available as the file ‘input-double.html’ as a separate download with D3 Tips and Tricks. A a copy of most the files that appear in the book can be downloaded (in a zip file) when you download the book from Leanpub.
<!DOCTYPE html>
<meta charset="utf-8">
<title>Double Input Test</title>

<p>
  <label for="nHeight" 
         style="display: inline-block; width: 240px; text-align: right">
         height = <span id="nHeight-value"></span>
  </label>
  <input type="range" min="1" max="280" id="nHeight">
</p>

<p>
  <label for="nWidth" 
         style="display: inline-block; width: 240px; text-align: right">
         width = <span id="nWidth-value"></span>
  </label>
  <input type="range" min="1" max="400" id="nWidth">
</p>

<script src="http://d3js.org/d3.v3.min.js"></script>
<script>

var width = 600;
var height = 300;
 
var holder = d3.select("body")
      .append("svg")
      .attr("width", width)    
      .attr("height", height); 

// draw a rectangle
holder.append("rect")
    .attr("x", 300)
    .attr("y", 150)
    .style("fill", "none")
    .style("stroke", "blue")
    .attr("height", 150) 
    .attr("width", 200);

// read a change in the height input
d3.select("#nHeight").on("input", function() {
  updateHeight(+this.value);
});

// read a change in the width input
d3.select("#nWidth").on("input", function() {
  updateWidth(+this.value);
});

// update the values
updateHeight(150);
updateWidth(100);

// Update the height attributes
function updateHeight(nHeight) {

  // adjust the text on the range slider
  d3.select("#nHeight-value").text(nHeight);
  d3.select("#nHeight").property("value", nHeight);

  // update the rectangle height
  holder.selectAll("rect") 
    .attr("y", 150-(nHeight/2)) 
    .attr("height", nHeight); 
}

// Update the width attributes
function updateWidth(nWidth) {

  // adjust the text on the range slider
  d3.select("#nWidth-value").text(nWidth);
  d3.select("#nWidth").property("value", nWidth);

  // update the rectangle width
  holder.selectAll("rect")
    .attr("x", 300-(nWidth/2)) 
    .attr("width", nWidth);
}

</script>
THE EXPLANATION
For the sake of brevity, this explanation will simply concentrate on the differences between the previous single input example and this one.
The declarations for the inputs in the HTML at the start of the code are simply duplicates of each other in terms of function;
<p>
  <label for="nHeight" 
         style="display: inline-block; width: 240px; text-align: right">
         height = <span id="nHeight-value"></span>
  </label>
  <input type="range" min="1" max="280" id="nHeight">
</p>

<p>
  <label for="nWidth" 
         style="display: inline-block; width: 240px; text-align: right">
         width = <span id="nWidth-value"></span>
  </label>
  <input type="range" min="1" max="400" id="nWidth">
</p>
The only significant difference is the declaration of the id’s for each input and it’s respective label.
The JavaScript selection of the inputs is more duplication;
d3.select("#nHeight").on("input", function() {
  updateHeight(+this.value);
});

d3.select("#nWidth").on("input", function() {
  updateWidth(+this.value);
});
Again the only substantive difference is the use of the appropriate id values.
The updating of the width and height is done via two different functions;
function updateHeight(nHeight) {

  // adjust the text on the range slider
  d3.select("#nHeight-value").text(nHeight);
  d3.select("#nHeight").property("value", nHeight);

  // update the rectangle height
  holder.selectAll("rect") 
    .attr("y", 150-(nHeight/2)) 
    .attr("height", nHeight); 
}

// Update the width attributes
function updateWidth(nWidth) {

  // adjust the text on the range slider
  d3.select("#nWidth-value").text(nWidth);
  d3.select("#nWidth").property("value", nWidth);

  // update the rectangle width
  holder.selectAll("rect")
    .attr("x", 300-(nWidth/2)) 
    .attr("width", nWidth);
}
The rectangle is selected using a common rect designator, so multiple rectangles could be controlled. But each function controls only a specific attribute (height or width).

Rotate text with an input

This example is really just a derivative of the adjustment of a single attribute of an element.
I happen to think it’s just a little bit ‘neater’ because it includes text, but in reality, it’s just another attribute that can be adjusted.
Here we let our range input adjust the rotation of a piece of text.
Text rotation with an input

THE EXPLANATION
We’ll dispense with the full code listing since it’s just a regurgitation of the adjusting of the radius of the circle example, but the code for the example is available online at bl.ocks.org or GitHub. It is also available as the file ‘input-text-rotate.html’ as a separate download with D3 Tips and Tricks. A a copy of most the files that appear in the book can be downloaded (in a zip file) when you download the book from Leanpub.
The only, thing of even a slight difference (other than some naming conventions) is the initial drawing of the text…
holder.append("text")
  .style("fill", "black")
  .style("font-size", "56px")
  .attr("dy", ".35em")
  .attr("text-anchor", "middle")
  .attr("transform", "translate(300,150) rotate(0)")
  .text("d3noob.org");
… and the update function;
function update(nAngle) {

  // adjust the text on the range slider
  d3.select("#nAngle-value").text(nAngle);
  d3.select("#nAngle").property("value", nAngle);

  // rotate the text
  holder.select("text") 
    .attr("transform", "translate(300,150) rotate("+nAngle+")");
}

Use a number input with d3.js

There are obviously different inputs that can be selected. The following example still rotates our text, but uses anumber type of input to do it;
<p>
  <label for="nValue" 
         style="display: inline-block; width: 240px; text-align: right">
         angle = <span id="nValue-value"></span>
  </label>
  <input type="number" min="0" max="360" step="5" value="0" id="nValue">
</p>
we have set the step value to speed things up a bit when rotating, but it’s completely optional.
The input itself can be adjusted up or down using a mouse click or have a number typed into the input box.
Text rotation with a number input
This type of input is slightly different from the range type since it isn’t fully supported under Firefox and as a result when I was testing it the arrow keys for going up and down weren’t present.
The full code for the example is available online at bl.ocks.org or GitHub. It is also available as the file ‘input-number-text.html’ as a separate download with D3 Tips and Tricks. A a copy of most the files that appear in the book can be downloaded (in a zip file) when you download the book from Leanpub.

Change more than one element with an input

The final example looking at using HTML inputs with d3.js incorporates a single input acting or two different elements. This might seem self evident, but if you’re as unfamiliar with HTML as I am (it’s embarrassing I know, but what can you do?) it may be of assistance.
The end result is to produce a single slider as a range input that rotates two separate text objects in different directions simultaneously.
Dual text rotation
THE CODE
The following is the full code for the example. A live version is available online at bl.ocks.org or GitHub. It is also available as the file ‘input-text-rotate-2.html’ as a separate download with D3 Tips and Tricks. A a copy of most the files that appear in the book can be downloaded (in a zip file) when you download the book from Leanpub.
<!DOCTYPE html>
<meta charset="utf-8">
<title>Input test</title>

<p>
  <label for="nAngle" 
         style="display: inline-block; width: 240px; text-align: right">
         angle = <span id="nAngle-value"></span>
  </label>
  <input type="range" min="0" max="360" id="nAngle">
</p>

<script src="http://d3js.org/d3.v3.min.js"></script>
<script>

var width = 600;
var height = 300;
 
var holder = d3.select("body")
      .append("svg")
      .attr("width", width)    
      .attr("height", height); 

// draw d3.js text
holder.append("text")
  .attr("class", "d3js")
  .style("fill", "black")
  .style("font-size", "56px")
  .attr("dy", ".35em")
  .attr("text-anchor", "middle")
  .attr("transform", "translate(300,55) rotate(0)")
  .text("d3.js");

// draw d3noob.org text
holder.append("text")
  .attr("class", "d3noob")
  .style("fill", "black")
  .style("font-size", "56px")
  .attr("dy", ".35em")
  .attr("text-anchor", "middle")
  .attr("transform", "translate(300,130) rotate(0)")
  .text("d3noob.org");

// when the input range changes update the rectangle 
d3.select("#nAngle").on("input", function() {
  update(+this.value);
});

// Initial starting height of the rectangle 
update(0);

// update the elements
function update(nAngle) {

// adjust the range text
  d3.select("#nAngle-value").text(nAngle);
  d3.select("#nAngle").property("value", nAngle);

  // adjust d3.js text
  holder.select("text.d3js") 
    .attr("transform", "translate(300,55) rotate("+nAngle+")");

  // adjust d3noob.org text
  holder.select("text.d3noob") 
    .attr("transform", "translate(300,130) rotate("+(360 - nAngle)+")");
}

</script>
THE EXPLANATION
The explanation for this example differes from the others in the way that the d3.js elements (the two pieces of text) are initially appended and then updated.
When they are initially drawn…
holder.append("text")
  .attr("class", "d3js")
  .style("fill", "black")
  .style("font-size", "56px")
  .attr("dy", ".35em")
  .attr("text-anchor", "middle")
  .attr("transform", "translate(300,55) rotate(0)")
  .text("d3.js");

holder.append("text")
  .attr("class", "d3noob")
  .style("fill", "black")
  .style("font-size", "56px")
  .attr("dy", ".35em")
  .attr("text-anchor", "middle")
  .attr("transform", "translate(300,130) rotate(0)")
  .text("d3noob.org");
… both elements are declared with a class attribute that serves as a reference for the future updating. Here, the text ‘d3.js’ is given a class name of d3js and the text ‘d3noob.org’ is given a class name of d3noob.
Then when we call the update function each of the two text elements is adjusted seperately by selecting each based on the class name that was applied in the initial setup;
function update(nAngle) {

// adjust the range text
  d3.select("#nAngle-value").text(nAngle);
  d3.select("#nAngle").property("value", nAngle);

  // adjust d3.js text
  holder.select("text.d3js") 
    .attr("transform", "translate(300,55) rotate("+nAngle+")");

  // adjust d3noob.org text
  holder.select("text.d3noob") 
    .attr("transform", "translate(300,130) rotate("+(360 - nAngle)+")");
}
So the ‘d3.js’ text is selected using text.d3js and ‘d3noob.org’ is selected using text.d3noob. That’s a pretty neat trick and a good lesson for applying specific transformations to specific objects.

The description above (and heaps of other stuff) is in the D3 Tips and Tricks book that can be downloaded for free (or donate if you really want to :-)).