Raspberry Pi Pico Tips and Tricks

Saturday 12 July 2014

Leanpub, d3.js, scruffy graphs and predicting the future

One of the neat things about using computers is learning new tricks.

Mostly the tricks are not new to the World, but they're new to me, so it's like the excitement of invention with a small side order of resigned disappointment.

But this doesn't make the journey any less interesting, so the following is a tale of doin' stuff for no particular reason other than wanting to learn new tricks and findin' stuff out that you didn't know you wanted to know, but it's kinda interesting anyway.

I like Leanpub. Its a great idea run by a motivated team who are providing an excellent service. If you're thinking that you might want to write a book, check them out. They make the process easy.

I have a book on Leanpub called 'D3 Tips and Tricks'. Hell, I have two books. The others imaginatively called 'Leaflet Tips and Tricks', but that's still a significant work in progress. You can download them for free if you want (d3.js is free, and I'm not clever enough to contribute to the code, so why not information?). D3 Tips and tricks is over 580 pages long, so it's amassed a bit of information since I started writing.

It turns out that d3.js is a pretty popular JavaScript library. People like what it can do and want to learn how to use it. So a book on d3.js that you can download for free from Leanpub is a an attractive proposition with little to no downside. It gets downloaded a fair bit and that's great. That's what it's there for. Help people out if they are looking for some information.

You have to have goals in life right? Whether it's something to aspire to or something you want to achieve (and make sure you know the difference) they're good to have.

When I started writing D3 Tips and Tricks the goal was to learn about d3.js and to take usable notes for myself that might be useful for others.  That worked well for me and (hopefully) well for those downloading the book. So that goal got ticked off as achieved.

Then I started to take notice of the number of people who were downloading the book. Let's get things straight, it's not a landslide, we're not talking `50 shades of SVG' (although Mike Bostock's probably got that covered with some of the work he's done on palettes). So numbers of downloads of the book has been part of my goals.

First I looked forward to just moving up the list of Lifetime Number of Copies Sold. One at a time. Slowly clawing my way upward. I kept little notes. Tried to get a feel for how the numbers around me were moving.

Then I realized that I might have a chance to break into the big time. Get on the front page so to speak. That meant getting into the top 10. I don't remember how many downloads that needed at the time, but today it would take about 5500. Time passed and that goal was met. Then I dared to dream that I might get onto the top shelf. Into the top five most downloaded books on Leanpub.

This was difficult going. books up there have a lot of downloads under their belt and a lot of momentum. But more time passed and D3 Tips and Tricks stumbled onto the top shelf.

Currently It's at number four and I'm finding it difficult to give myself goals to relate to the number of people who download the book since there's not too much wiggle room to improve. But then I realized that the question I had to ask myself was how much wiggle room do I have?

So in the interests of learning a new trick, I worked out how to scrape the page of a Leanpub book to extract the number of downloads that it had and I started to record the number of downloads of the top 10 books on Leanpub every day. Using that information I started to develop a scruffy graph to show how D3 Tips and Tricks was doing in relation to it's peers.

This graph shows how many more or less books are sold than D3 Tips and Tricks of the books near the top of the most downloaded list.

It's tricky to read the text on the right because of the compressed nature of the books, but here's the key to identifying who is who;

bt = Backbone Tutorials
tnbb = The Node Beginner Book
hon = Hands-on Node.js
d3tat = D3 Tips and Tricks
htdwyl = How to do what you love and earn what you're worth
watir = Web Application Testing in Ruby
lh = Lisp Hackers
taopwi = The ABC of PDF with iText
qgffoad = Quick guide for Firefox OS App Development

There's very little to see there really other than to notice that D3 Tips and Tricks is doing fairly well compared to most, but 'The ABC of PDF with iText' (the blue line) is doing really well.

This graph didn't really help me with my goals, but then I realized that I really needed to look into the future to see what might become of all of these books and their interactions.

So I made a nice graph of trends.

That provided me with a better view of the downloading landscape and I can see that there are some significantly different gradients in there.

The ABC of PDF with iText has some serious upwards movement and has been carving up the competition at a ferocious rate. By the looks of it D3 Tips and Tricks won't be too far away from being a notch on it's belt.

But when? How will this play out? Well, the lines are fairly consistent, if we take the length of time that the data has been captured for and extrapolate using the number of books downloaded in that time we can see how the rankings and downloads will vary into the future.

Now we're talking. There's some interesting interactions.

This gave me a chance to play with a new trick I had just learned with d3.js to invert the position of the mouse on the screen into a representative date, so I was able to make a graph that provided a read out of the date that something would happen and the number of books downloaded when you hovered your mouse over it.

Sure you could do this manually, but I wouldn't have learned anything then!

So, feel free to go here to see a graph of the upcoming action.

From this, here are the predictions for the next few months.

  • 24 August 2014:  Quick guide for Firefox OS App Development will pass Lisp Hackers at around 6000 downloads.
  • 25 October 2014: Web Application Testing in Ruby will pass How to do what you love and earn what you're worth at around 8380 downloads.
  • 2 February 2015: D3 Tips and Tricks will pass Hands-on Node.js at around 17480 downloads.
  • 2 March 2015: Quick guide for Firefox OS App Development will pass How to do what you love and earn what you're worth at around 8380 downloads.
  • 7 March 2015: The ABC of PDF with iText will pass Hands-on Node.js at around 17970 downloads.
  • 14 March 2015: D3 Tips and Tricks will pass The Node Beginner Book at around 18590 downloads.
  • 26 March 2015: The ABC of PDF with iText will pass The Node Beginner Book at around 18770 downloads.
  • 5 April 2015: The ABC of PDF with iText will pass D3 Tips and Tricks at around 18770 downloads.
(Edit 22-Jul-2014: I originally typo'd the first two predictions (swapped the passing books) :-( so they read correctly (per the graph) now)

From this D3 Tips and Tricks can hope for a climb to the lofty heights of 2nd place, but this will be short lived as The ABC of PDF with iText will force it back to 3rd after only a couple of weeks.

Clearly none of this is written stone. There are so many variables I haven't considered that it makes my head spin. What sort of variations over time will each book face. Will there be a bolter from below (the astute among you will note that Laravel 4 Official Documentation doesn't feature on the graph. I can't remember why it's not there. It's possible that it wasn't reporting the number of readers when I developed the scraper)?

But what would be the point of doing all this graphical stuff if you weren't having a little fun :-).

Friday 11 July 2014

My favourite tooltip method for a line graph

The following post is a portion of the D3 Tips and Tricks book which is free to download. To use this post in context, consider it with the others in the blog or just download the the book as a pdf / epub or mobi .

My Favorite tooltip method for a line graph.


Tooltips are a fabulous way to include an interactive element on a graph and a great mechanism for including additional, focused information to the user.
There are quite a number of different ways to implement tooltips (one of which you can find in the ‘Adding tooltips’ section of the ‘Assorted Tips and Tricks’ chapter of D3 Tips and Tricks) and I would be very hesitant about proclaiming any one better than another. However, the one we will work through here is my favorite when using a line graph as I think it brings a ‘fuzzier’ mechanism for deciding when a tooltip is highlighted (you don’t have to be over an object to get information on it) which I like.
I believe that the original example for this was shown by Mike Bostock here, but I first came across the technique in an example by ‘gniemetz’. I liked ‘gniemetz’s example enough to adapt a similar example which I will explain below.
The idea with this technique is to set an area the size of the graph that will be used to determine when a tooltip will be displayed. So that when the mouse enters that area, the display style that allows elements to be shown or hidden. This then tells the script to show the tooltip and the location of the mouse determines which point will have the tooltip. In the example below we can see that the mouse cursor is some distance away from the point that is being highlighted, but it is in line (in the vertical axis) with the highlighted point (in fact we will use some clever maths to determine which date point (or point on the x axis) is the one that will be used to generate the tooltip.
More complicated favorite tooltip example
To begin this explanation we’ll start with a simple example that will just project a circle on the point where the tooltip will appear. Once we’ve worked out how that works we can add whatever we want and I will explain what is going on in the more complex example.
As mentioned, we will start with a simple example that adds a circle on the point where we will place our tooltip. It will look a bit like this;
Simple version of the favorite tooltip example

The Code

The full code for this simple example is available online at bl.ocks.org or GitHub. It is also available as the files ‘best-tooltip-simple.html’ and ‘atad.csv’ as a download with the book D3 Tips and Tricks (in a zip file) when you download the book from Leanpub.
I have placed commented out asterisks besides the lines that have been added or altered from the simple graph example that we started out with at the beginning of the book so that it’s easy to see what has changed.
<!DOCTYPE html>
<meta charset="utf-8">
<style> /* set the CSS */

body { font: 12px Arial;}

path { 
    stroke: steelblue;
    stroke-width: 2;
    fill: none;

.axis path,
.axis line {
    fill: none;
    stroke: grey;
    stroke-width: 1;
    shape-rendering: crispEdges;


<!-- load the d3.js library -->    
<script src="http://d3js.org/d3.v3.min.js"></script>


// Set the dimensions of the canvas / graph
var margin = {top: 30, right: 20, bottom: 30, left: 50},
    width = 600 - margin.left - margin.right,
    height = 270 - margin.top - margin.bottom;

// Parse the date / time
var parseDate = d3.time.format("%d-%b-%y").parse;
    bisectDate = d3.bisector(function(d) { return d.date; }).left; // **

// Set the ranges
var x = d3.time.scale().range([0, width]);
var y = d3.scale.linear().range([height, 0]);

// Define the axes
var xAxis = d3.svg.axis().scale(x)

var yAxis = d3.svg.axis().scale(y)

// Define the line
var valueline = d3.svg.line()
    .x(function(d) { return x(d.date); })
    .y(function(d) { return y(d.close); });
// Adds the svg canvas
var svg = d3.select("body")
        .attr("width", width + margin.left + margin.right)
        .attr("height", height + margin.top + margin.bottom)
              "translate(" + margin.left + "," + margin.top + ")");

var lineSvg = svg.append("g");                             // **********

var focus = svg.append("g")                                // **********
    .style("display", "none");                             // **********

// Get the data
d3.csv("atad.csv", function(error, data) {                 // **********
    data.forEach(function(d) {
        d.date = parseDate(d.date);
        d.close = +d.close;

    // Scale the range of the data
    x.domain(d3.extent(data, function(d) { return d.date; }));
    y.domain([0, d3.max(data, function(d) { return d.close; })]);

    // Add the valueline path.
    lineSvg.append("path")                                 // **********
        .attr("class", "line")
        .attr("d", valueline(data));

    // Add the X Axis
        .attr("class", "x axis")
        .attr("transform", "translate(0," + height + ")")

    // Add the Y Axis
        .attr("class", "y axis")

    // append the circle at the intersection               // **********
    focus.append("circle")                                 // **********
        .attr("class", "y")                                // **********
        .style("fill", "none")                             // **********
        .style("stroke", "blue")                           // **********
        .attr("r", 4);                                     // **********
    // append the rectangle to capture mouse               // **********
    svg.append("rect")                                     // **********
        .attr("width", width)                              // **********
        .attr("height", height)                            // **********
        .style("fill", "none")                             // **********
        .style("pointer-events", "all")                    // **********
        .on("mouseover", function() { focus.style("display", null); })
        .on("mouseout", function() { focus.style("display", "none"); })
        .on("mousemove", mousemove);                       // **********

    function mousemove() {                                 // **********
        var x0 = x.invert(d3.mouse(this)[0]),              // **********
            i = bisectDate(data, x0, 1),                   // **********
            d0 = data[i - 1],                              // **********
            d1 = data[i],                                  // **********
            d = x0 - d0.date > d1.date - x0 ? d1 : d0;     // **********

        focus.select("circle.y")                           // **********
            .attr("transform",                             // **********
                  "translate(" + x(d.date) + "," +         // **********
                                 y(d.close) + ")");        // **********
    }                                                      // **********




You should be able to tell from the asterisks in the code above that there aren’t too many changes and appart from a few at the start and middle, the majority are contained in a large block towards the end.
Starting with our first change
    bisectDate = d3.bisector(function(d) { return d.date; }).left;
This is our function that will be called later in the code that returns a value in our array of data that corresponds to the horizontal position of the mouse pointer. Specifically it returns the date that falls to the left of the mouse cursor.
The d3.bisector is an ‘array method’ that can use an accessor or comparator function to divide an array of objects. In this case our array of date values. In the code I have used the d3.bisector as an accessor, because I believe that it’s simpler to do so for the point of explanation, but the downside is that I had to have my dates ordered in ascending order which is why I load a slightly different csv file later (atad.csv).
If your eyes glazed over slightly reading the previous paragraph, don’t let that put you off. Like with so many things, just relax and let d3.js do the magic and remember that d3.bisector can find a value in an ordered array.
The next block of changes declares a couple of functions that we will use to add our elements to our graph;
var lineSvg = svg.append("g");

var focus = svg.append("g")
    .style("display", "none");
We will use lineSvg to add our line for the line graph and focus will add our tooltip elements. it is possible to avoid using lineSvg, but this way of declaring the functions means that we can control which elements are on top of which on the screen. For instance, it would be a pretty sad affair if our tooltip was appearing under the line of the line graph (hard to read).
As we saw earlier, our data is being sourced from a different csv file (atad.csv).
d3.csv("atad.csv", function(error, data) {
This is because we need to have it in a compatible order (ascending) to allow our bisector function to operate correctly. So while the line may look the same as the simple graph version, the data is ordered in reverse (some may say that this is the way the original data should have been presented all along, but I suppose we can’t always second guess the data we get).
We then make a small change to the script that appended the line to the graph and instead of usingsvg.append… we use our newly declared lineSvg.
        .attr("class", "line")
        .attr("d", valueline(data));
The final, larger block of code can be broken into 4 logical sections;
  1. Adding the circle to the graph
  2. Set the area that we use to capture our mouse movements
  3. The clever maths that determines which date will be highlighted
  4. Move the circle to the appropriate position
The last two points actually occur within a separate function, but for the purposes of explanation I’m happy that this is a logical division of labour for the script.
Adding the circle to the graph is actually fairly simple;
        .attr("class", "y")
        .style("fill", "none") 
        .style("stroke", "blue")
        .attr("r", 4);
If you’ve followed any of the other examples in D3 Tips and Tricks there shouldn’t be any surprises here (well, perhaps assigning a class to the circle (y) could count as mildly unusual).
Except for one small thing….
We don’t place it anywhere on the graph! There is no x y coordinates and no translation of position. Nothing! Never fear. All we want to do at this stage is to create the element. In a few blocks of code time we will move the circle.
As we briefly covered earlier, the thing that makes this particular tooltip technique different is that we don’t hover over an element to highlight the tooltip. Instead we move the mouse into an area which is relevant to the tooltip and it appears.
And its all thanks to the following code;
        .attr("width", width)
        .attr("height", height)
        .style("fill", "none")
        .style("pointer-events", "all")
        .on("mouseover", function() { focus.style("display", null); })
        .on("mouseout", function() { focus.style("display", "none"); })
        .on("mousemove", mousemove);
Here we’re adding a rectangle to the graph (svg.append("rect")) with the same height and width as our graph area (.attr("width", width) and .attr("height", height)) and we’re making sure that there’s no colour (fill) in it (.style("fill", "none")). Nothing too weird about all that.
Then we make sure that if any mouse events occur within the area that we capture them (.style("pointer-events", "all")). This is when things start to get interesting.
The first pointer event that we want to work with is mouseover;
        .on("mouseover", function() { focus.style("display", null); })
This line of code tells the script that when the mouse moves over the area of the rectangle of the area of the graph the display properties of the focus elements (remember that we appended our circle to focus earlier) are set to null. This might sound like a bit of a strange thing to do, since what we want to do is to make sure that when the mouse moves over the graph we want the focus elements to be displayed. but by setting the displaystyle to null the default value for display is enacted and this is inline which allows the elements to be rendered as normal. So why not use inline instead of null? Good question. I’ve tried it and it works without problem, but the original example that Mike Bostock used had the setting at null and I’ll make the assumption that Mike knows something that I don’t know about when to use null and when to use inline for a display style (maybe some browser incompatibility issues?).
The reverse of making our focus element display display everything is being able to make it stop displaying everything. This is what happens in the next line;
        .on("mouseout", function() { focus.style("display", "none"); })
Here, where the mouse moves off the area, the display properties for the focus element are turned off.
Lastly for this block, we need to capture the actions of the mouse as it moves on the graph area and move our tooltips as required. This is accomplished with the final line in the block…
        .on("mousemove", mousemove);
… where if the mouse moves we call the mousemove function.
Once the mousemove function is called is carries out the last two steps in our code. The first of which is the clever maths that determines which point in our graph has the tooltip applied to it.
  var x0 = x.invert(d3.mouse(this)[0]),
      i = bisectDate(data, x0, 1),
      d0 = data[i - 1],
      d1 = data[i],
      d = x0 - d0.date > d1.date - x0 ? d1 : d0;
The first line of this block is a dozy;
  var x0 = x.invert(d3.mouse(this)[0]),
If we break it down the d3.mouse(this)[0] portion returns the x position on the screen of the mouse (d3.mouse(this)[1] would return the y position). Then the x.invert function is reversing the process that we use to map the domain (date) to range (position on screen). So it takes the position on the screen and converts it into an equivalent date!
for the adventurous amongst you, throw a console.log(x0); line into the mousemove function and check out the changing date/time as the cursor moves pixel by pixel (This will work for Google Chrome). Very cool.
Then we use our bisectDate function that we declared earlier to find the index of our data array that is close to the mouse cursor.
      i = bisectDate(data, x0, 1),
It takes our data array and the date corresponding to the position of or mouse cursor and returns the index number of the data array which has a date that is higher than the cursor position.
Then we declare arrays that are subsets of our data array;
      d0 = data[i - 1],
      d1 = data[i],
d0 is the combination of date and close that is in the data array at the index to the left of the cursor and d1 is the combination of date and close that is in the data array at the index to the right of the cursor. In other words we now have two variables that know the value and date above and below the date that corresponds to the position of the cursor.
The final line in this segment declares a new array d that is represents the date and close combination that is closest to the cursor.
      d = x0 - d0.date > d1.date - x0 ? d1 : d0;
It is using the magic JavaScript short hand for an if statement that is essentially saying if the distance between the mouse cursor and the date and close combination on the left is greater than the distance between the mouse cursor and the date and close combination on the right then d is an array of the date and close on the right of the cursor (d1). Otherwise d is an array of the date and close on the left of the cursor (d0).
This could be regarded as a fairly complicated little piece of code, but if you take the time to understand it, you will be surprised how elegant it appears. As we’ve seen before though, if you just want to believe that the d3.js magic is happening, that’s fine.
The final block of code that we’ll check out takes the closest date / close combination that we’ve just worked out and moves the circle to that position;
            "translate(" + x(d.date) + "," +  
                           y(d.close) + ")"); 
This is a pretty easy bit of code to follow. We select the circle (using the class y that we assigned to it earlier) and then move it using translate to the date / close position that we had just worked out was the closest.
Of course this is provision of the coordinates to the circle that we noticed was missing earlier in the code when we were appending it to the graph.
And there we have it. A simple circle positioned at the closest point to the mouse cursor when the cursor hovers over the graph.
Simple version of the favorite tooltip example
If we hadn’t mentioned it earlier you might be thinking that this could possibly be the most complicated method for making most basic (read lame) tooltip ever. But you know there’s more right? Right….? Read on.

Complex version

You’ve read to this point, so that’s a sign that you’re still interested. In that case, I recommend that you take a moment to check out the live example of the graph that I’m going to describe.
More complicated favorite tooltip example
Here’s a graph that when you move your mouse over it shows the closest intersection point on the graph with lines that extend the full width of the graph (great for comparing the level across the graph) and down to the x axis (to get a rough feel for the date). As well as this there is a subtle circle around the data point in question (as already explained in the previous section) and the actual date and value represented at the intersection point. As if that wasn’t enough there is a nice little drop shadow effect under the text so that no matter what the background is you can read it. Nice.
The full code for this example is available online at bl.ocks.org or GitHub. It is also available as the files ‘best-tooltip-coolio.html’ and ‘atad.csv’ as a download with the book D3 Tips and Tricks (in a zip file) when you download the book from Leanpub.
Because the date at the tooltip needs to be formatted in a particular way we need to declare this appropriately;
    formatDate = d3.time.format("%d-%b"),
Other than that everything is pretty normal until we get to the part where we start adding elements to our focusgroup (you remember we had the circle before? Now we’re adding additional elements.).
   // append the x line
        .attr("class", "x")
        .style("stroke", "blue")
        .style("stroke-dasharray", "3,3")
        .style("opacity", 0.5)
        .attr("y1", 0)
        .attr("y2", height);

    // append the y line
        .attr("class", "y")
        .style("stroke", "blue")
        .style("stroke-dasharray", "3,3")
        .style("opacity", 0.5)
        .attr("x1", width)
        .attr("x2", width);

    // append the circle at the intersection
        .attr("class", "y")
        .style("fill", "none")
        .style("stroke", "blue")
        .attr("r", 4);

    // place the value at the intersection
        .attr("class", "y1")
        .style("stroke", "white")
        .style("stroke-width", "3.5px")
        .style("opacity", 0.8)
        .attr("dx", 8)
        .attr("dy", "-.3em");
        .attr("class", "y2")
        .attr("dx", 8)
        .attr("dy", "-.3em");

    // place the date at the intersection
        .attr("class", "y3")
        .style("stroke", "white")
        .style("stroke-width", "3.5px")
        .style("opacity", 0.8)
        .attr("dx", 8)
        .attr("dy", "1em");
        .attr("class", "y4")
        .attr("dx", 8)
        .attr("dy", "1em");
Here you can see we’re adding the x (horizontal) line and the y (vertical) line as well as the date and text values. Notice on the text values, there is a white drop shadow added first and then the text over the top. Another thing to note is that just like the position information, we don’t actually put the text in here, this is simple a ‘placeholder’ for the element.
Then all we need to do is move all the new elements to the correct position and add the changing text where appropriate;
            "translate(" + x(d.date) + "," +
                           y(d.close) + ")");

            "translate(" + x(d.date) + "," +
                           y(d.close) + ")")

            "translate(" + x(d.date) + "," +
                           y(d.close) + ")")

            "translate(" + x(d.date) + "," +
                           y(d.close) + ")")

            "translate(" + x(d.date) + "," +
                           y(d.close) + ")")

            "translate(" + x(d.date) + "," +
                           y(d.close) + ")")
                 .attr("y2", height - y(d.close));

            "translate(" + width * -1 + "," +
                           y(d.close) + ")")
                 .attr("x2", width + width);
There’s no big surprises here. Just an extension of what we accomplished with the circle earlier. The only part that looks semi-interesting is some of the application of the positioning of the x and y lines and this is more because of the points at which the lines start and finish.
Now this is unlikely to be the end solution for most people, but at least there are plenty of examples of different elements in there to play with and experiment on.

The description above (and heaps of other stuff) is in the D3 Tips and Tricks book that can be downloaded for free (or donate if you really want to :-)).