I’m currently working on my next book for Three.js and one of the chapters deals with visualizing open data. When looking around for data that I could use, I ran across a set of data from the NOAA. Through this site you can download a set of monthly precipiation reports for the whole world in a gridded format. So I downloaded them, and started playing around with the data to see how it looks and how it could be used. In this article I’m not going to show you the three.js based result, but I’ll give you a quick overview how to get to the format I initially used for debugging purposes:
In this image you can see the monthly precipitation for the whole world, on a logarithmic scale for July 2012. I’ve also created a simple site that shows this, and the animation , in action.
So, what do you need to do to convert the set you can download from the NOAA site to something visually.
- Download and convert the NetCDF format.
- Load in the resulting CSV file
- Process the CSV data into a world grid
- Animate the transitions between the months
- As a bonus: also create a legend to show what color means what
First, though, we need to get the data.
Download and convert the NetCDF format
The first thing we need to do is get the data. I’ve used the following link: where you can define the range of data you want to download. For this example I’ve used the range from January 2012 to December 2012 and selected the option to create a subset without making a plot.
The format in which it is downloaded, however, isn’t directly usable as input to our HTML5 canvas based visualization. You can use ncdump-json to create a JSON file, but you still need to be able to interpret it, so I choose the alternative way. I just wrote a simple java program to convert the NetCDF format to a simple CSV file.
I’ve used the following maven dependencies:
<dependencies>
<dependency>
<groupId>edu.ucar</groupId>
<artifactId>netcdf</artifactId>
<version>4.2.20</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.4</version>
</dependency>
</dependencies>And use the following piece of java code:
public class NetCDFDump {
public static void main(String[] args) throws IOException, InvalidRangeException {
String year = "2012";
NetcdfFile nc = NetcdfFile.open("src/main/resources/X84.31.143.145.44.1.47.49.nc");
Variable precip = nc.findVariable("precip");
// use the shapes to create an array
int[] shapes = precip.getShape();
// month, lat, lon
float[][][] data = new float[shapes[0]][shapes[1]][shapes[2]];
// iterate over 12 (or 11) months
int[] pos = new int[3];
int[] shape = {1,1,1};
for (int i = 0 ; i < shapes[0] ; i++) {
pos[0]=i;
for (int lat = 0 ; lat < shapes[1]; lat++) {
pos[1] = lat;
for (int lon = 0 ; lon < shapes[2]; lon++) {
pos[2] = lon;
Array result = precip.read(pos, shape);
data[pos[0]][pos[1]][pos[2]] = result.getFloat(0);
}
}
}
// output data like this
// month, lat, lon, humidity
float[][] combined = new float[data[0].length][data[0][0].length];
for (int m = 0 ; m < data.length ; m++) {
File outputM = new File(year + "-out-" + m + ".csv");
for (int lat = 0 ; lat < data[m].length ; lat++) {
for (int lon = 0 ; lon < data[m][lat].length; lon++) {
float value = data[m][lat][lon];
if (value > -1000) {
combined[lat][lon]+=value;
} else {
combined[lat][lon]+=-1000;
}
// write the string for outputfile
StringBuffer bOut = new StringBuffer();
bOut.append(m);
bOut.append(',');
bOut.append(lat);
bOut.append(',');
bOut.append(lon);
bOut.append(',');
bOut.append(value);
bOut.append('\n');
// write to month file
FileUtils.write(outputM,bOut,true);
}
}
}
// now process the combined
File outputM = new File(year + "-gem.csv");
for (int i = 0; i < combined.length; i++) {
for (int j = 0; j < combined[0].length; j++) {
StringBuffer bOut = new StringBuffer();
bOut.append(i);
bOut.append(',');
bOut.append(j);
bOut.append(',');
bOut.append(combined[i][j]/data.length);
bOut.append('\n');
FileUtils.write(outputM, bOut, true);
}
}
}
}I won’t go into too much detail what’s happening, but this piece of code results in a number of files, one for each month, and one containing the average.
Each month is shown in the following format
... 0,65,78,32.65 0,65,79,35.09 0,65,80,31.14 0,65,81,42.7 0,65,82,49.57 ...
Respectively the values mean: the month, latitude, longitude and precipitation. For the average it looks almost the same, except the first entry is omitted.
... 59,94,59.874165 59,95,65.954994 59,96,57.805836 ...
Now that we’ve got the data in an easy to use format, we can use it to create the visualizations.
Load in the resulting CSV file
To load the file we just use a simple XMLHttpRequest like this:
// create an XMLHttpRequest to get the data
var xmlhttp = new XMLHttpRequest();
xmlhttp.onreadystatechange = function() {
if (xmlhttp.readyState == 4 && xmlhttp.status == 200) {
var coords = CSVToArray(xmlhttp.responseText,",");
// and process each of the coordinates
...
}
}
// make the call and use the callback to process the result
xmlhttp.open("GET", "location/of/the/file", true);
xmlhttp.send();The coords variable now contains all the coordinates, and for each coordinate the value to show. Converting this to a canvas is actually very easy.
Process the CSV data into a world grid
In the callback from the XMLHttpRequest we check whether we’ve received the data and convert it to a set of coordinates. The only thing we need to do is convert these coordinates to a visualization on the canvas.
var coords = CSVToArray(xmlhttp.responseText,",");
coords.forEach(function(point) {
var offset = 0;
if (point.length > 3) {
offset = 1;
}
if (parseFloat(point[2+offset]) >= 0) {
var lat = parseInt(point[0+offset]);
var lon = parseInt(point[1+offset]);
var value = parseFloat(point[2+offset]);
if (value > max) max = value;
// lat is from 0 to 180
// lon is from 0 to 360
var x = canvas.width/360*((lon)-180);
if (x<=0) {
x=canvas.width-(x*-1);
}
var y = canvas.height/180*lat;
if (value >= 0) {
context.beginPath();
context.rect(x,y,4,4);
context.fillStyle = scale(value).hex();
context.fill();
}
}
});As you can see, very simple code where we just take the positions, convert them to an X and Y coordinate on the canvas, and create a small square with a specific color. For the generation of the color we use a Chroma.js scale.
var scale = chroma.scale(['red' , 'yellow', 'green', 'blue']).domain([1,1700], 100, 'log');
This call creates a color scale from red to yellow to green to blue. The values range from 1 to 1700, are divided into 100 steps, and use a logarithmic scale. This results in the following image (this time for the precipitation in January 2012:
Since we’ve got figures for all the months, we can now easily create a simple animation.
Animate the transitions between the months
For animations we’re going to create something like shown in the following movie, where we transition slowly between the various months:
Creating this animation can be done rather easy by just showing the images on top of each other, and changing the opacity. So first set up some css, which hides most of the images and puts them all one on top of each other.
#cf {
position:relative;
margin:0 auto;
height: 700px;
}
#cf img {
position:absolute;
left:0;
width: 1600px;
}Now we can just add the images and using the ‘bottom’ class to only show the first image:
<div id="cf">
<img id="img-1" class="top" src="./assets/images/2012-01-perc.png" />
<img id="img-2" class="bottom" src="./assets/images/2012-02-perc.png" />
<img id="img-3" class="bottom" src="./assets/images/2012-03-perc.png" />
<img id="img-4" class="bottom" src="./assets/images/2012-04-perc.png" />
<img id="img-5" class="bottom" src="./assets/images/2012-05-perc.png" />
<img id="img-6" class="bottom" src="./assets/images/2012-06-perc.png" />
<img id="img-7" class="bottom" src="./assets/images/2012-07-perc.png" />
<img id="img-8" class="bottom" src="./assets/images/2012-08-perc.png" />
<img id="img-9" class="bottom" src="./assets/images/2012-09-perc.png" />
<img id="img-10" class="bottom" src="./assets/images/2012-10-perc.png" />
<img id="img-11" class="bottom" src="./assets/images/2012-11-perc.png" />
<img id="img-12" class="bottom" src="./assets/images/2012-12-perc.png" />
</div>Now we just need some javascript to tie everything together:
var month=[];
month[0]="January";
month[1]="February";
month[2]="March";
month[3]="April";
month[4]="May";
month[5]="June";
month[6]="July";
month[7]="August";
month[8]="September";
month[9]="October";
month[10]="November";
month[11]="December";
var allTweens;
init();
animate();
function init() {
// create a chain of tweens
allTweens = setupTweens(12);
allTweens[0].start();
}
function setupTweens(imageCount) {
var tweens = [];
for (var i = 0 ; i < imageCount ; i++) {
var tween = new TWEEN.Tween( { opac: 0, image: i, max: imageCount } )
.to( { opac: 100 }, 2500 )
.easing( TWEEN.Easing.Linear.None )
.onUpdate( function () {
// on update, lower the opacity of image i and update the opacity of
// image i+1;
var currentImage = document.getElementById('img-'+(this.image+1));
if (this.image == imageCount -1) {
var nextImage = document.getElementById('img-'+1);
} else {
var nextImage = document.getElementById('img-'+(this.image+2));
}
currentImage.style.opacity = 1- this.opac / 100;
nextImage.style.opacity = this.opac / 100;
} );
tween.onComplete(function() {
document.getElementById('title-2012').textContent = "Showing precipitation: " + month[this.image] + " " + 2012;
// Set the inner variable to 0.
this.opac = 0;
// we're done, restart
if (this.max-1 == this.image) {
allTweens[0].start();
}
});
// connect to each another
if (i > 0) {
tweens[i-1].chain(tween);
}
tweens.push(tween);
tweens[0].repeat();
}
return tweens;
}
function animate() {
requestAnimationFrame(animate);
TWEEN.update();
}Here we use tween.js to setup the transitions between the images.
As a bonus: also create a legend to show what color means what
In the animation you can see a legend at the bottom. This legend was created as a simple canvas, saved as a image. For completeness sake, the code to do this is shown here:
var canvas = document.createElement("canvas");
canvas.width = 435;
canvas.height = 30;
var context = canvas.getContext('2d');
var domains = scale.domain();
document.body.appendChild(canvas);
// from 1 to 1700
for (var i = 0 ; i < domains.length ; i++) {
context.beginPath();
context.rect(10+i*4,0,4,20);
console.log(domains[i]);
context.fillStyle = scale(domains[i]).hex();
context.fill();
}
context.fillStyle = 'black';
context.fillText("0 mm", 0, 30);
context.fillText(Math.round(domains[25]) + " mm", 100, 30);
context.fillText(Math.round(domains[50]) + " mm", 200, 30);
context.fillText(Math.round(domains[75]) + " mm", 300, 30);
context.fillText("1700 mm", 390, 30);Here we just use the scale we’ve seen easier and walk through the domains to create a colored legend.
