Example: Link NLCD using Scan Raw Binary File

In this example we use the Scan Raw Binary File tool to scan and to prepare a configuration file, which we use to link an NLCD raw binary file providing land cover data for Delaware as a raster image.   We use a standard palette to color the land cover data and then we assign a projection to the newly imported image so it can be used as a correctly georegistered layer in maps.   This process is tedious because, despite the use of tools like Scan Raw, binary files do not include modern conveniences such as automatic specification of file organization, palette and coordinate system, so those must be specified manually.

 

We begin by choosing Tools- Scan Raw Binary File

 

 

We click the [...] browse button to navigate to the raw binary file we will use.

 

 

We have downloaded files from a USGS FTP site.   The delaware.bin file contains raw binary data for a raster image that provides NLCD (National Land Cover Data) information for the state of Delaware in the form of a palette image.   The other files are text files which provide documentation and commentary.   We click the delaware.bin file and press Open.

 

 

The dialog automatically loads a recommended name to use for the .rwb configuration file that will be created.   Option boxes are enabled with default values.

Find Necessary Information

The most difficult part of importing raw binary or raw text files usually is the detective work required to find out how the files are structured.  Important details are buried, at times using unusual terminology, within various text files.   If we are lucky, such commentary will accompany the data file.   If we are unlucky, we may have to dig through endless pages on web sites and undertake Internet searches to find the required documentation.

 

This example shows a case where we get lucky, and all of the information we require is written down within the text files that accompanied the data file.   That the required info is buried deep within endless bureaucratic commentary, requiring us to scroll though many pages of text is typical.   Who said GIS was is for the faint of heart?

 

 

Scrolling through the FGDC metadata text we discover the information highlighted above.   That tells us the data is an 8-bit integer (since we are using a binary file and not a GeoTIFF) and that it is one channel (single band).    We also note the comment on projection information for later use.

 

Many pages further down into the text we see the information highlighted above.   That tells us the Size of the raster image.   3300 is the width or X extent and 5170 is the height or Y extent.   We see from the above that the metadata provides important information about the coordinate system, such as the use of Albers Conical Equal Area projection, which we will use later in this example.

 

 

We choose int8 in the Type box, enter [3300, 5710] in the Size box and choose 1 for the number of Channels.   We have not seen any commentary about extra bytes in the beginning of the file or the use of any padding bytes, nor is there any documentation for NULL values, so we will leave those options in their default settings.   We press Scan to try a scan.

 

 

The scan appears to work OK.   There are 0 extra values, so it appears we have the Size of the raster, the number of Channels, and the number of bytes in the Type correct. If those factors were incorrect the scan would have reported extra values.  We press Save to create an .rwb configuration file.

 

Link Data

Our next step is to link the raw binary data into our project.  We choose File - Link.

 

 

In the Link dialog we click onto the delaware.bin.rwb configuration file and choose Open.

 

 

That creates a new data source in the Project.   We open the data source, Ctrl-click on the delaware image and the delaware Tiles table to highlight them, and then we press Ctrl-C to Copy them.

 

 

We click into the main part of the project to move the focus there, that is, outside of the data source, and then we press Ctrl-V to Paste a copy of the image and the image's table into the main part of the project.   We can then click the delaware.bin data source to highlight it and press Delete to delete it since we no longer need the linked data source.  We now have a purely local copy of the image in our project.   

 

We could have, of course, continued using the delaware image from within the linked data source.  However, .map files are much faster than binary files so it makes sense to save a local copy of the image into a .map project file.

 

 

Double-click the delaware image to open it.   It appears using typical default formatting for a palette image.  The black zone around the edges of the data is where pixel values are 0.    We will now style the image to use a standard NLCD palette.

 

 

In the Style pane we choose Channel 0 as the source, we use unique values as the method and then we add the above intervals and colors.   This takes forever to do one-by-one using the * button as described in the Example: Add, Delete and Edit Thematic Formatting Intervals topic.  The Notes section below gives a shortcut way of assigning the intervals using only a few clicks.

 

Press Update Style.

 

 

We now see Delaware using a typical NLCD palette, where each color means a particular land cover classification.  The black region that does not contain data, for which pixel values are 0, is not seen because we have styled it using transparent color.   See the discussion in the Style: Invisible Pixels topic.

 

 

Each color has a meaning in this image.   Red, for example, means regions which are "medium intensity developed", as seen in the zoomed-in view above showing New Castle airport.   See the Notes for a legend of colors and meanings.

Assign a Projection

The above is wonderful but has limited usefulness if a coordinate system is not correctly assigned, so the data can be used as a layer in maps with other data.   Raw binary files do not convey projection information.   We must specify the coordinate system in use manually, based upon what we read in the documentation for the data file.    Manifold, of course, has a dialog for that.

 

 

Clicking the Component pane we see the coordinate system for the delaware image is shown in red color as an unconfirmed projection, using the default Pseudo Mercator projection for placeholder text.    To assign the correct initial coordinate system we click on the coordinate system picker button.

 

 

We choose Assign Initial Coordinate System and then Edit Coordinate System.   If the data used one of our favorites, we could do the job in one click but, ...no such luck today!

 

 

In the Coordinate System dialog's Custom tab we choose Albers Conical Equal Area as the Type of coordinate system, which we remember from reading the documentation.  Now, the hard part: we must find all the right numbers to enter into the various parameter boxes.   To help us, we will us information from the delaware.txt and delaware_fgdc.txt files:

 

 

The delaware_fgdc.txt file has most of what we need, as we saw earlier in this topic.    But it is missing one key piece of information that we must get from the delaware.txt file.

 

 

We must open the delaware.txt file in Vim, an open source text editor, because the delaware.txt file uses UNIX style end of line codes that result in impossible-to-read text in Windows Notepad.   From this file we get the upper left and lower right corner coordinates for the projection in use.   These we will use to calculate Local offset X and Y values.

 

 

Based on what we read in the text files, we load the boxes in our dialog as seen above, using Copy and Paste from the documentation text files whenever possible, to avoid typographical errors.   

 

 

We also click into the metrics picker button to enter values into the Coordinate System Metrics dialog, as seen above.   We then click OK, and OK again, to surface from both dialogs.

 

The Local scale values came from the Abscissa_Resolution and Ordinate_Resolution values, both 30, in the delaware_fgdc.txt file.   The Local offset values came from the Upper Left Corner X value and the Lower Right Corner Y value in the delaware.txt file, subtracting 15 from both.   We do that because the data set is centered on the middle of a 30 x 30 size pixel, but Manifold aligns rasters to the lower left edge.  Therefore, we must subtract half a pixel's worth of meters, 15, from the local offsets to match alignment in Manifold with the alignment used by the raw data.    

 

Is all that an incredible amount of annoying manual labor?  Of course.  That is why GIS in modern times avoids use of raw data formats which do not encapsulate and automatically provide such information when we import data.

 

Press OK.   The delaware image now has the correct coordinate system assigned and can be used as a layer in maps with other data.

 

 

We create a new map, drop a Bing streets image server background layer into it, and then drop the delaware image into the map.   The map automatically re-projects on the fly the Albers Conical Equal Area projection used by the image into the Pseudo-Mercator projection used by Bing and by the map.  Delaware appears where it should, at the correct scale and orientation.

 

 

Zooming in we can see the data set is correctly aligned.  

 

Notes

In the example above we mentioned a short cut to provide a Style palette that saves the time of specifying intervals and colors manually.   We do that by copying the necessary JSON string from this topic, and then pasting it into the StylePixel property value for the delaware image in the mfd_meta table.     To learn how that works, please begin by reading the Example: Add, Delete and Edit Thematic Formatting Intervals topic.

 

Next, use Style to assign any convenient, even a default, style to the delaware image using Channel 0.   For example, use Channel 0 as the source and press Update Style for whatever appears as a default.   That creates a StylePixel property value for the delaware image in the mfd_meta table.  

 

 

If we open the mfd_meta table we can find the StylePixel property value for the delaware image.   That property is selected in the illustration above so it stands out in red selection color for a clearer illustration - there is no need to actually select rows in the mfd_meta table to copy and paste from them.   We can right-click onto the Value cell for the StylePixel property and Copy, Paste or Edit whatever is in the cell.   In the illustration above we choose Edit.

 

 

The editing dialog shows the full JSON string for the long list of NLCD intervals and colors illustrated earlier in this topic.   We can Copy that text and paste it below, into this topic:

 

{ "Channel": 0, "Value": -16777216, "Values": { "0": -16777216, "11": 4746658, "12": 15200252, "21": 14798286, "22": 16122112, "23": 11403525, "24": 11206913, "31": 13485245, "32": 11053224, "33": 4929899, "41": 7121254, "42": 1926451, "43": 12438675, "51": 11573312, "52": 13745026, "61": 5792040, "71": 15592653, "72": 13685121, "73": 10800209, "74": 8567453, "81": 14538814, "82": 11432489, "83": 14329688, "84": 12040622, "85": 15703094, "91": 12310509, "92": 7055048 } }

 

To use the above, simply Copy the text above and then Paste it into the StylePixel value cell in the mfd_meta table for the image which is to be colored using that palette.   The palette provides a combination of values from the 1992 NLCD classifications (as used in the delaware image example) and in addition includes some intervals that image does not use, taken from the more recent update to NLCD colors seen below.   Because that palette was tediously created for us by Manifold as part of the creation of this topic, we do not need to manually create that palette.   We can simply copy and paste the text above.

 

 

When creating palettes manually, if we want to copy some published scheme, the color picker tool that is available when double-clicking into a color well is invaluable.  If we have an image or a web site that shows a palette arrangement as above, we leave that open somewhere on our Windows desktop and then we can copy colors that appear in the arrangement using the color picker tool.

 

See Also

Component Pane

 

Style

 

Style: Invisible Pixels

 

Style: Palettes

 

File - Import

 

File - Link

 

Tools - Scan Raw Binary / Text File

 

Coordinate System

 

Assign Initial Coordinate System

 

Example: Add, Delete and Edit Thematic Formatting Intervals - This topic provides a step by step example of adding, deleting and editing intervals in the Style dialog that are used for thematic formatting.

 

Example: Style Properties in the mfd_meta Table - Style properties for drawings such as colors for areas are stored in human readable JSON values as properties in the mfd_meta system table.   This example shows how we can copy formatting from one drawing to another by simply copying values between records in the mfd_meta table.