Workflow Automation in Whitebox GAT

(This is part 1 of a multi-part post. See Workflow Automation Part 2 here.)

Okay, Whitebox GAT provides users with a very user-friendly means of accessing geoprocessing tools through dialog boxes with built-in help documentation. The dialog boxes are very consistent in design and the tools are each readily accessible from the Tools tab in the tool tree-view and the quick-search lists. This likely provides the most convenient way to access this functionality for most of your day-to-day work. Every now and then however you’ll find yourself in a situation where you have a workflow with numerous intermediate steps and you need to repeat those steps several times over. This is a case for automation. Every plugin tool in Whitebox can be run automatically  from a script. In fact, the help documentation for each tool provides a description of how that tool can be scripted. In the longer term I intend to develop a graphical workflow automator that will automatically generate scripts based on a workflow graph that the user creates. This type of functionality, similar to what is found in ArcGIS, Idrisi, and a few others, can be quite helpful for automating certain tasks. They are never as powerful however as directly scripting a workflow and that is why so many GIS users these days have become comfortable with scripting. It’s an extremely powerful tool for the GIS analyst to have.

Whitebox GAT currently provides scripting support for three programming languages including Python (actually Jython), Groovy, and JavaScript. I know that because ArcGIS supports Python scripting there are many GISers out there that are familiar with Python. Those of you who have been paying close attention to Whitebox development may have noticed that I have started to write a lot of new plugin tools using Groovy (most of Whitebox is developed using Java). Why Groovy instead of Python, you may ask? Well Groovy is a superset of Java, meaning that most valid Java is valid Groovy (if you know Java you already know Groovy). Groovy, being a scripting language, has some syntax advantages that make it much terser than regular Java. Compared with Jython, the Groovy project also seems to be more actively maintained and certainly has better computational performance, particularly with the optional static compilation. So for writing full-blown plugin tools, Groovy makes a lot of sense. (As an aside, how can you not like a language with such a great name?) But for simply automating workflows, I know that most of you are going to be looking at good old Python.

So, that said, below is a Python script that provides an example of how to automate a workflow. In this case, it automates the very common task in spatial hydrology of taking in a digital elevation model, removing the topographic depressions, calculating the flow directions, calculating the flow accumulation grid, then extracting streams. The script could be made much terser but I added a lot of comments to clarify. Simply paste the script into the Whitebox Scripter and press Execute Code. Scripting in Whitebox GAT can be a heck of a lot of fun and awfully addictive once you realize how much power you have at your fingertips! As always, best wishes and happy geoprocessing.

'''
Copyright (C) 2014 Dr. John Lindsay <jlindsay@uoguelph.ca>

This program is intended for instructional purposes only. The
following is an example of how to use Whitebox's scripting
capabilities to automate a geoprocessing workflow. The scripting
language is Python; more specifically it is Jython, the Python
implementation targeting the Java Virtual Machine (JVM).

In this script, we will take a digital elevation model (DEM),
remove all the topographic depressions from it (i.e. hydrologically
correct the DEM), calculate a flow direction pointer grid, use
the pointer file to perform a flow accumulation (i.e. upslope area)
calculation, then threshold the upslope area to derive valley lines
or streams. This is a fairly common workflow in spatial hydrology.

When you run a script from within Whitebox, a reference to the
Whitebox user interface (UI) will be automatically bound to your
script. It's variable name is 'pluginHost'. This is the primary
reason why the script must be run from within Whitebox's Scripter.

First we need the directory containing the data, and to set
the working directory to this. We will use the Vermont DEM contained
within the samples directory.
'''

import os

separator = os.sep # The system-specific directory separator
wd = pluginHost.getApplicationDirectory() + separator + "resources" + separator + "samples" + separator + "Vermont DEM" + separator
pluginHost.setWorkingDirectory(wd)

demFile = wd + "Vermont DEM.dep"
# Notice that spaces are allowed in file names. There is also no
# restriction on the length of the file name...in fact longer,
# descriptive names are preferred. Whitebox is friendly!

# A raster or vector file can be displayed by specifying the file
# name as an argument of the returnData method of the pluginHost
pluginHost.returnData(demFile)

'''
Remove the depressions in the DEM using the 'FillDepressions' tool.

The help file for each tool in Whitebox contains a section detailing
the required input parameters needed to run the tool from a script.
These parameters are always fed to the tool in a String array, in
the case below, called 'args'. The tool is then run using the 'runPlugin'
method of the pluginHost. runPlugin takes the name of the tool (see
the tool's help for the proper name), the arguments string array,
followed by two Boolean arguments. The first of these Boolean
arguments determines whether the plugin will be run on its own
separate thread. In most scripting applications, this should be set
to 'False' because the results of this tool are needed as inputs to
subsequent tools. The second Boolean argument specifies whether the
data that are returned to the pluginHost after the tool is completed
should be suppressed. Many tools will automatically display images
or shapefiles or some text report when they've completed. It is often
the case in a workflow that you only want the final result to be
displayed, in which case all of the runPlugins should have this final
Boolean parameter set to 'True' except for the last operation, for
which it should be set to 'False' (i.e. don't suppress the output).
The data will still be written to disc if the output are supressed,
they simply won't be automatically displayed when the tool has
completed. If you don't specify this last Boolean parameter, the
output will be treated as normal.
'''
filledDEMFile = wd + "filled DEM.dep"
flatIncrement = "0.001" # Notice that although this is a numeric parameter, it is provided to the tool as a string.
args = [demFile, filledDEMFile, flatIncrement]
pluginHost.runPlugin("FillDepressions", args, False, True)

# Calculate the D8 pointer (flow direction) file.
pointerFile = wd + "pointer.dep"
args = [filledDEMFile, pointerFile]
pluginHost.runPlugin("FlowPointerD8", args, False, True)

# Perform the flow accumulation operation.
flowAccumFile = wd + "flow accumulation.dep"
outputType = "number of upslope grid cells"
logTransformOutput = "False"
args = [pointerFile, flowAccumFile, outputType, logTransformOutput]
pluginHost.runPlugin("FlowAccumD8", args, False, True)

# Extract the streams
streamsFile = wd + "streams.dep"
channelThreshold = "1000.0"
backValue = "NoData"
args = [flowAccumFile, streamsFile, channelThreshold, backValue]
pluginHost.runPlugin("ExtractStreams", args, False, False) # This final result will be displayed

'''
Note that in each of the examples above, I have created new variables
to hold each of the input parameters for the plugin tools. I've done
this more for clarity than anything else. The script could be
substantially shorted if the shorter variables were directly entered
into the args array. For instance, I could have easily used:

args = [flowAccumFile, streamsFile, "1000.0", "NoData"]

for the last runPlugin and saved myself declaring the two variables.
Because the file names are generally used in subsequent operations,
it is a good idea to dedicate variables to those parameters.
'''
Whitebox Scripter

Whitebox Scripter

Edit
Catching Bugs Before They Bug You:

When you’re writing a script in Whitebox, if your program throws an error, the software will record the error in your log files. The log files are xml files contained within the logs directory within the main Whitebox folder. They are detailed printouts of exactly what was happening around the time that the exception was thrown. They can certainly be challenging to read. An easier way of dealing with this problem is to incorporate exception handling in your script. Here’s a brief example:

try:
	i = 14
	k = "24"
	j = i + k # Throws an error
except Exception, e:
	print e
	pluginHost.showFeedback("Error during script execution.")
	''' alternatively, you many want to send the exception to 
	the pluginHost.logException() method '''
finally:
	print "I'm done!"

This code will print the following statement to the Whitebox Scripter console:

TypeError(“unsupported operand type(s) for +: ‘int’ and ‘unicode'”,)

4 thoughts on “Workflow Automation in Whitebox GAT

  1. Simon Seibert says:

    Thanks John for putting so much effort in setting up Whitebox and for providing all these wonderful examples. They make Whitebox realy easy and straightforward to use. I ran into Whitebox a few weeks ago (by accident) and meanwhile I’m wondering how I managed to live without it in the past 😉

    I wrote a couple of (python) scripts for my one work (phd thesis). However I do not have a solid background in Python (so far). I would like to know if it possible to include (for)loops in the scripter as well? I would like to run the same script over many files. Could you also provide an example for such a problem?

    Thanks a lot
    Simon

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s