Video: Open GIS Data Portals

Open GIS data portals are becoming more common, and can contain lots of geospatial data, but is it a case of too much of a good thing?

What is a Portal?
An internet site providing access or links to other sites. In the case of open GIS data, the file may appear on the same page, link out to another website (or portal) or require more digging.  In addition, some GIS portals have links to web map services.

Search is the thing...
Open data portals house more and more data and therefore can become more difficult to search over time. Creating complex searches is sometimes not available and can slow the discovery process.

What are they made of?
Open data portals leverage free and open, propriety, or hybrid data and/or GIS platforms.

Here one minute, gone the next...
While often data can be downloaded directly, some sites link to external or partner data sources. Some data may seem accessible and near-at-hand but actually require more searching and digging.

Syndication
In some cases, this is the result of syndicated data sets. For example, on federal data portals, local and state governments can have their data sets metadata appear in search results and link out to the resource.  In others, this can be overused, frustrating, or lead to broken links or missing resources. Sometimes going directly to the source can be easier than navigating a much larger data portal.

Examples
We will look at several open data portals for geospatial data:

• Data.gov
• ESRI's Open Data
• National Weather Service GIS Data Portal
• NYC Open Data

Also check out http://dataportals.org/ for a neat interactive map of open data portals worldwide.

Looking for an open data portal?  Check out the interactive and searchable map on dataportals.org

More 3D Maps and Tips Using QGIS2threejs

After last week's popular post, here are few more maps and tips using the QGIS2threejs plugin.

The Maps

1. 3D Chloropleth Map of Crime Counts by Police Beats in Chicago, 2013
2. Visualization of Mt. Rainier, Seattle
3. Groundwater Depths in Three Counties in Nevada

Data sources and references:
Open Data Chicago
Chicago: Official Police Maps (CLEARMAP)
National Map Viewer
Nevada Bureau of Mines and Geology

When viewing the maps, if a black box appears instead of a basemap, try refreshing/reloading your browser and the basemap should display properly.

The Tips

3D Chloropleth Map:  Here is a trick to create a 3D chloropleth map on a flat surface.  The plugin won't run without a digital elevation model (or DEM) raster, but you can 'trick' it into using a flat raster.

• Updated(12/9/2015): The plugin now allows users to select a flat surface/no DEM used
• Take the raster calculator and exponentiate/raise all cell values to the power of 0 (rastervariable^0). The resulting raster will be flat with a value of 1.  Alternatively, multiply by zero.  (You may also have to symbolize null values as well.)
• Classify and symbolize the polygons using your desired color scheme--before using the plugin
• In addition, I changed the color of the raster file in QGIS to match the background in the *.html file (#ccccff). 
• I added HTML code for a basic legend using frames (90% for the map, 10% for the legend)

Basic HTML code sourcing the two frames.  The "Chicago" frame was generated from QGIS.
The "Legend" frame was a simple piece of code directed to a JPG of the Legend.

Screenshot of the final map and legend.

Mt Rainier:  Instead of using a web service, I download imagery and used the Build Virtual Raster in QGIS.   I used a detailed DEM and imagery from the USGS's National Map Viewer.  You can download files from USGS by using a bounding box, for example.  I used a vertical exaggeration of 1.1 in the plugin.  Click the screenshot below to enlarge a comparison of between Google Earth and the plugin's version of Mt. Rainier.

Left: Google Earth, Right: Plugin result using files from USGS

3D Groundwater Depths / Underground:  Using point data with depths, you can either select a negative multiplier in the plugin or create a new variable and multiply your depths by -1.

If your depths are positive values, select a negative multiplier in the plugin.

An overhead and underground view of the map.
The orange boundary represents three counties of interest.

3D Visualization of Manhattan using QGIS Plugin

GIS Blogger Anita Graser, aka Underdark, has created a nice visualization using the QGIS2threejs plugin, including an interactive web-based version.  I won't repeat the instructions listed on her blog, but I will make a few suggestions as I go along. Her eloquent map features buildings extruded (constant height), trees, and some elevation changes. Feeling inspired, I thought I would give it a quick try.

If you just want to see the visualization, visit: http://webmapexamples.net/3DExample/Cityscape.html

Note: You will need Firefox, Chrome, or Internet Explorer 11--basically a Web-GL capable browser to view the visualizations.

WebGL is an API that allows for 3D visualizations to be displayed in a web browser without additional plugins. So, users won't need to install anything to view your product.  The threejs plugin is a JavaScript library that helps to keep everything neat and tidy.  For examples of threejs projects, check out this website: http://threejs.org/ which is filled with examples of featured projects.  Believe it or not, the projects you will see below consist of five files that take up less than 5 megabytes of storage.

Tip #1: Using a shapefile or digitizing a boundary for an extent can help to speed the project along and keep the look of your map consistent.  In QGIS, you can do this using the AutoTrace plugin.

For the project, I focused on Manhattan from Central Park down to just below times square.

Below is what the project looks like in QGIS.  I left the bound on to show the project area.  I used a building footprint shapefile that contained height from New York City's Open Data Portal.  Please note: This file is large ~ 180 MB.  In addition, there is a basemap from Open Street Map using the QGIS plugin, and a DEM.  You can also add other point, polygon, or line features to your 3D map.  I have chosen to stay with just buildings.  I color-coded the buildings based on height for effect (redder is taller).  Click on any of the screenshots to magnify them.

Project area and files 

Running the plugin is rather quick.  The result is five files exported to a folder of your choosing.  The contents of the folder can be placed into the root or public directory of web hosting service to publish to the web!  Of course, you can also double-click the resulting *.html file to open it on your desktop--please note some broswers may try to block you from opening it.  So right-click, and choose open with a friendly browser like Chrome or Firefox.

Tip #2:  I created a new variable to half the building heights.  Otherwise, the tall buildings of NYC barely fit in the web browser and makes viewing awkward and challenging.

You can view the visualization at: http://webmapexamples.net/3DExample/Cityscape.html

When you first open the html file, the view will be oriented the same way you have the files in QGIS.  If you have not used 3D viewers before, you can navigate using the left and right mouse buttons and wheel to  pan, rotate, and zoom.  Try not to make sudden moves with the mouse because it is easy to get disoriented. Refreshing the web page will send you back to the starting view.

3D Visualization of Manhattan.  Redder buildings are taller.

As you zoom and pan, you can bring areas into closer view.  Here looking down 7th Avenue in NYC.

A closer look: From Central Park down 7th Avenue

Lastly, an overhead view of Times Square...

Can you see where the ball is dropped on New Year's...One Times Square?

Overall, the plugin was fast and easy to use.  Not only can it be used for creating cityscapes...You could also visualize chloropleth maps in 3D, for example.  Performance in browser was smooth.  I saw memory usage similar to what Underdark reported, around 1.6 GB of RAM.  This is reduced if you do not constantly hold down any of the mouse buttons--occasionally releasing as you view.

Update #1: If the basemap is not loading, and you see a black rectangle, reload/refresh your browser once or twice and it should appear!

Update #2: Forgot to mention: Since almost 2 GB of RAM is used--this prevents viewing the maps on a mobile device, since most mobile devices don't have or max out at 2 GB.  I have gotten them to load in Android after a long time, but you won't be able to manipulate them (zoom, pan, rotate,etc.)