Whitebox GAT for ArcGIS users

A short time ago over on the Whitebox GAT listserv someone asked an excellent question. They asked whether there was such thing as a Whitebox GAT User’s Manual, since the transition to Whitebox GAT, when you’re coming from an ArcGIS background, can be a little difficult. Often there are corresponding tools in Whitebox GAT to those in ArcGIS but they will frequently (and sometimes frustratingly, I’d imagine) have different names. There are different workflows for doing similar tasks and in general you can feel a little like a fish out of water for a short while until you get use to the Whitebox GAT way of doing things.

I can certainly understand this perspective. To date, I’ve been the main contributor to the Whitebox GAT project and so it reflects my design decisions more than anything. And, let’s face it, I’m a little different. As I pointed out to the question asker, I have never in my past had a time when I have used ArcGIS on a day-to-day basis. I’ve used IDRISI extensively many years ago (and that fact is apparent in some aspects of Whitebox in the way certain things are done) but frankly I’m just not that familiar with ArcGIS. It doesn’t inform how I have developed Whitebox to a large extent.

While we’re still working on a draft of a comprehensive Whitebox GAT User’s Manual (thank you to Adam Bonnycastle for his efforts here), I have been busily compiling quick tutorials on common tasks in Whitebox, which I have been cleverly embedding in the Help menu:

Whitebox's Help menu

Whitebox’s Help menu

Notice, that list starting with ‘Getting the Whitebox source code’? I’m not sure to what extent this feature has been taken up, but it’s there. If you have any suggestions for other additions let me know either by emailing me or leaving a comment below. If you have any Whitebox GAT tutorials that you’ve created yourself and would like to donate to the project, also please let me know.

But this gets me to the main point of this blog. What I would like to do is offer a brief tutorial on the topic “Whitebox GAT for ArcGIS users“. It can be loosely formed as a list of common gottcha’s or ‘Them vs. Us’ comparisons. But as I’ve said above, I really don’t use ArcGIS, so I need to ask for your help. If you’re a Whitebox GAT user and are coming to it from an ArcGIS background, can you provide us with a list of the main difficulties you experienced when you first tried using Whitebox? We’re looking for some of the things that you had to spend time trying to figure out how to do. Wouldn’t it be great if you could help to make that transition a little easier for future Whitebox users? I’m a big believer in the idea of paying it forward. Leave your comments below and, as always, best wishes and happy geoprocessing.

Advertisements

Whitebox Geospatial Analysis Tools v. 3.2.1 Now Released!

I am very pleased to announce the release of version 3.2.1 of the free and open-source GIS Whitebox Geospatial Analysis Tools. Although I was not able to accomplish all of the development tasks that I had set out to do over the summer, this version does incorporate numerous improvements and additions (see below for details). Whitebox 3.2.1 requires Java 8u20 or higher, which can be downloaded from the Oracle site here. Please update your Java runtime before launching Whitebox. I hope that you enjoy the new version. As always, I welcome any feedback you may have, including suggestions for future features and tools and feedback on general usability. Bugs and other errors can be submitted using the ‘Report An Error‘ found within Whitebox’s Help menu. For questions regarding specific applications, please consider subscribing to the Whitebox Listserv which can be found at the following site:

http://www.lsoft.com/scripts/wl.exe?SL1=WHITEBOX-GAT&H=LISTSERV.UOGUELPH.CA

The following modifications have been made:

  • Added the Retrieve SRTM DEM Data tool. This tool will download SRTM DEM tiles from the USGS FTP site, import the tiles into Whitebox GAT, and optionally fill missing data holes and mosaic the tiles.
  • Added the ability to natively display LAS file point clouds in the map area.
  • Added Hex-binning tool for producing hexagonally gridded heat-maps (this is pretty cool for data visualization!).
  • Added a tool for isolating ground return points from LiDAR LAS files. I plan on adding additional methods for performing this type of LiDAR point filtering in the near future.
  • Added a tool for separating large LiDAR LAS files into smaller tiles of shapefile points.
  • Added a Lee filter (Sigma filter) tool.
  • Added Interior Point tool.
  • Added CreateRectangularGridTool for creating a vector grid of polygons based on
    a rectangular grid.
  • Added Single-part to Multi-part and Multi-part to Single-part tools.
  • Added List Unique Values tool to list all of the unique values in a categorical
    field contained within a vector’s attribute table. It will also report category
    frequency.
  • Added a fast depression breaching tool. Depression breaching is a far superior
    method for pre-processing a DEM for hydrological analysis (e.g. mapping
    watersheds). One of the reasons that people continue to fill depressions instead of breaching is that filling is computationally much more efficient than
    breaching. Well this tool is just about as fast as Whitebox’s depression filling
    tool. So now you have no excuse. This should be your default tool for hydrological processing of DEMs. Its result is not as good as the optimal breaching tool but it is far more efficient and still much better than filling. Please breach!
  • Added the tool BurnStreamAtRoads, which will carve a stream path through road embankments at the site of road/stream crossings.
  • Added a Minimum Interpolation tool for shapefile point inputs similar to the
    Minimum Interpolation (LiDAR) tool.
  • Added the ability to update individual scripts from the source code repository.
    This way, if you modify the code for a tool and break it, you can always
    revert it back to the source on the main trunk. There’s also now a menu entry
    that will update all scripts for which there are newer versions within the
    code repository and new scripts.
  • Added a Flood Order tool; details in tool’s help.
  • Added a polygonize tool for converting polylines into polygons.
  • Updated the centroid (vector) tool with the option to handle multi-part polyline
    and polygon features as a single entity or to extract centroids for each part,
    as well as, the ability to extract centroids for groups of points.
  • Created a convenience tool, Feature Selection, for opening the feature selection tab within the attribute table dialog after selecting a vector layer. If the layer is not currently displayed, it will be. The reason I added this tool is
    because many people search for feature selection in the toolbox rather than
    in the attribute table dialog.
  • The Hillshade tool will now automatically calculate an appropriate z conversion value when it detects that the DEM is in geographic coordinates with XY units of degrees. I also fixed the Slope, Aspect, and all the Curvature tools to do this as well.

Workflow Automation Part 2

In my earlier post on workflow automation in Whitebox, Simon Seibert left a comment asking, “I would like to know if it possible to include for loops in the Scripter as well? I would like to run the same script over many files. Could you also provide an example for such a problem?” Well Simon, yes you can. Here is an example of a simple Python script that finds all of the raster files contained within the current working directory and then performs a very simple analysis on each file:

import os
# The following code will find each raster
# file (.dep) in the working directory and
# then run a mean filter on the image.
wd = pluginHost.getWorkingDirectory()
a = 1
for file in os.listdir(wd):
  if file.endswith(".dep"):
    inputFile = wd + file
    outputFile = wd +"output" + str(a) + ".dep"
    a += 1
    xDim = "3"
    yDim = "3"
    rounded = "false"
    reflectEdges = "true"
    args = [inputFile, outputFile, xDim, yDim, rounded, reflectEdges]
    pluginHost.runPlugin("FilterMean", args, False)

If you want to take it to the next level, you can parallelize the script so that each iteration is run on a separate thread. So, there you have it. Leave your comments below and, as always, best wishes and happy geoprocessing.

Hexbinning in Whitebox GAT

The practice of binning point data to form a type of 2D histogram, density plot, or what is sometimes called a heatmap, is quite useful as an alternative for the cartographic display of of very dense points sets. This is particularly the case when the points experience significant overlap at the displayed scale. This is the type of operation that is used (in feature space) in Whitebox‘s Feature Space Plot tool. Normally when we perform binning, however, it is using a regular square or perhaps rectangular grid. The other day I read an interesting blog on hexbinning (see also the excellent post by Zachary Forest Johnson also on the topic of hexbinning), or the use of a hexagonal-shaped grid for the creation of density plots.

A Hex-binned heatmap

A Hex-binned heatmap (click to enlarge)

These blogs got me excited, in part because I have used hex-grids for various spatial operations in the past and am aware of several advantages to this tessellation structure. The linked hexbinning blogs point out that hexagonal grids are useful because the hexagon is the most circular-shaped polygon that tessellates. A consequence of this circularity is that hex grids tend to represent curves more naturally than square grids, which includes better representation of polygon boundaries during rasterization. But there are several other advantages that make hexbinning a worthwhile practice. For example, one consequence of the nearly circular shape of hexagons is that hex-grids are very closely packed compared with square grids. That is, the average spacing between hexagon centre points in a hex-grid is smaller than the equivalent average spacing in a square grid. One way of thinking about this characteristic is that it means that hexagonal grids require about 13% fewer data points then the square grid to represent a distribution at a comparable level of detail. Also, unlike a square grid, each cell in a hex-grid shares an equal-length boundary with its six neighbouring grid cells. With square grids, four of the eight neighbouring cells are connected through a point only. This causes all sorts of problems for spatial analysis, not the least of which is the characteristic orientation sensitivity of square grids; and don’t get me started on the effects of this characteristics for surface flow-path modelling on raster digital elevation models. Hex-grid cells are also equally distant to each of their six neighbours. I’ve even heard it argued before that given the shape of those cone and rod cells in our eyes, hex-grids are more naturally suited to the human visual system, although I’m not sure how true this is.

Hexagonal grids are certainly worthwhile data structures and hex-binning is a great way to make a heatmap. So, that said, I decided to write a tool to perform hex-binning in Whitebox GAT. The tool will be publicly available in the 3.2.1 release of the software, which will hopefully be out soon but here is a preview:

Whitebox's Hexbinning tool

Whitebox’s Hexbinning tool (click to enlarge)

The tool will take either an input shapefile of POINT or MULTIPOINT ShapeType, or a LiDAR LAS file and it will be housed in both the Vector Tools and LiDAR Tools toolboxes. Here is an example of a hex-binned density map (seen on right) derived using the tool applied to a distribution of 7.7 million points (seen on left) contained in a LAS file and derived using a terrestrial LiDAR system:

Hexbinned LiDAR points

Hexbinned LiDAR points (Click to enlarge)

Notice that the points in the image on the left are so incredibly dense in places that you cannot effectively see individual features; they overlap completely to form blanket areas of points. It wouldn’t matter how small I rendered the points, at the scale of the map, they would always coalesce into areas. The hex-binned heatmap is a far more effective way of visualizing the distribution of LiDAR points in this case.

The hexbinning tool is also very efficient. It took about two seconds to perform the binning on the 7.7 million LiDAR points above using my laptop. In the CUNY blog linked above, the author (I think it was Lee Hachadoorian) describes several problems that they ran into when they performed the hexbinning operation on their 285,000 points using QGIS. I suspect that their problems were at least partly the result of the fact that they performed the binning using point-in-polygon operations to identify the hexagon in which each point plotted. Point-in-polygon operations are computationally expensive and I think there is a better way here. A hex-grid is essentially the Voronoi diagram for the set of hexagon centre points, meaning that every location within a hexagon grid cell is nearest the hexagon’s centre than another other neighbouring cell centre point. As such, a more efficient way of performing hexbinning would be to enter each hexagon’s centre point into a KD-tree structure and perform a highly efficient nearest-neighbour operation on each of the data points to identify their enclosing hexagon. This is a fast operation in Whitebox and so the hexbinning tool works well even with massive point sets, as you commonly encounter when working with LiDAR data and other types of spatial data.

So I hope you have fun with this new tool and make some beautiful maps. Leave your comments below and, as always, best wishes and happy geoprocessing.

John Lindsay