LiDAR technology (Light Detection and Ranging) measures terrain elevations by shining a pulsed laser from a sensing platform onto a surface and measuring reflected pulses. Manifold imports or links LiDAR data from LAS or LAZ format files as points in a drawing, using whatever coordinate system is specified in the file. Manifold's LAS dataport can also Export the drawing's table to LAS or LAZ format files. A related capability is using the librarylas dataport to create a data source that generates a drawing from an entire folder, or hierarchy of folders, containing multiple LAS or LAZ files. See the librarylas topic for details.
Please see the Importing and Linking topic for the difference between linking a file into a Manifold project and importing data from that file into a Manifold project.
For users familiar with LiDAR data and LAS and LAZ formats, here are a few quick tips:
Install large data storage - Fast, large, hard disks are cheap. A six TB hard disk that can store ten thousand uncompressed LiDAR files of typical, 600MB size in LAS format is around $130 in the beginning of 2020. A two TB SSD is about $260, storing over 3,300 uncompressed LiDAR files. Get plenty of storage space.
Use LAS files, not LAZ files - LAS files are significantly faster than LAZ format because there is no delay from decompression. LAS files are usually ten times bigger than LAZ, but given the absurdly low cost of large data storage, who cares? LiDAR data is often published in LAZ format to reduce download sizes. Get a copy of the free LASzip package and use LASzip to decompress all LAZ files into LAS files as soon as you download them, so they'll be ready to use when you want. Don't save LiDAR data as LAZ files. Use LAS files for more speed.
Link LAS files, don't import them - Manifold builds a high performance spatial index when we link a LAS or LAZ file, but not when we import it. Future builds will add the high performance index for imported files as well, but until then it is faster to link a LAS or LAZ file into a project.
Nothing happens at first... but just once - The very first time we open a drawing within a linked LAS or LAZ data source, the drawing window will pop open but we will see nothing. That is because Manifold is building the spatial index in background. When the spatial index is built, the drawing window will automatically populate with the LiDAR points. The index is stored in an accessory file next to the LAS or LAZ file, so any future uses of that file in Manifold will have no delays to build an index. The drawing will pop open and be populated right away.
LAS file format is an ASPRS (American Society for Photogrammetry and Remote Sensing) standard format for storing LiDAR data as points. It is a vector format, not a raster format. LAZ format is identical to LAS except for being compressed. This documentation uses the term LAS to mean both LAS and LAZ formats, except when addressing performance or size issues that are specific to either LAS or LAZ.
LAZ format is popular for publishing LiDAR data because LAZ files are usually many times smaller than LAS files. A 450 MB LAS file, for example, will often compress to a LAZ file that is well under 60 MB. The price we pay for smaller file sizes is that for general purpose GIS work LAZ format must be decompressed to recover the original LAS data, at least into memory cache or temporary working files. Decompression is fast, but nonetheless significant time, possibly tens of seconds per file, is required to decompress bigger LAZ files.
Therefore, for faster opening and initial use of LiDAR data in Manifold, it is quicker to use uncompressed LAS files than compressed LAZ files. To decompress LAZ files into LAS form, we can use the LASzip.exe utility to decompress many LAZ files into LAS format.
Spatial indices allow far faster work with spatial data, but spatial indices designed for general GIS work are not efficient with the very large numbers of individual point objects in a typical LiDAR data set. Manifold therefore builds a specialized index for point cloud data when linking to LAS or LAZ files individually or when using the librarylas dataport to create a data source linked from a folder containing LAS or LAZ files.
The point cloud spatial index is built the first time a LAS or LAZ file is linked into a Manifold project. That can take significant time, possibly minutes per file in the case of LAZ files (faster with LAS files), but the spatial index needs only to be created once. The point cloud spatial index that is created is stored in an accessory .mapindexp file in the same folder as the linked LAS or LAZ file. Manifold will automatically use that .mapindexp spatial index the next time the accompanying LAS or LAZ file is used in a project.
Whenever we copy to another location LAS or LAZ files that have been used with Manifold, we also should always copy along with them the accompanying .mapindexp (point cloud spatial index) and .mapmeta (coordinate system) accessory files Manifold has created, so those accessory files can be re-used for much faster launching of projects that use their LAS or LAZ files.
The specialized point cloud index is a recent addition to Manifold which has not yet been implemented for LAS or LAZ files that are imported into a Manifold project. That creates, for now, the unusual situation that Manifold can display and work with LiDAR data faster when linking a LAS or LAZ file into a project, than importing the file into the Manifold project. Usually, workflow is much faster when we import a file into Manifold; however, LiDAR point cloud data in LAS files, for now, is an exception to that general rule.
Upcoming Manifold builds will include the specialized point index within .map files as well, which will mean imported LiDAR data will be even faster than linked LiDAR files. However, for now we should link LiDAR files if we want maximum speed. A bonus of linking LiDAR data into a project is that the Manifold project stays small, typically only one megabyte in size, since all of the LiDAR data remains stored in the LAS file.
Manifold starts building a point cloud index when we first open either a linked LAS or LAZ drawing or the drawing's table. The moment we click open the table, or we click open the drawing or add the drawing as a layer to an opened map, Manifold will launch a background process to build the index. Building the index in background allows us to keep working, doing other things, while we wait for the index to be built. Although the drawing window will open, no points will be visible until the index is finished building.
Build 220.127.116.11, the first build that runs index-building in a background thread, has a quirk: when we first open a drawing for a LAS or LAZ file that does not have a spatial index accessory file, the map window will report that there is no spatial index, and it will suggest building a temporary one. Do not do that. Instead, wait until the spatial index being built in background is ready (a minute or two) and refresh the drawing using View - Refresh. Builds after 18.104.22.168 ensure that map windows do not offer to build spatial indexes in such cases.
The Manifold LAS dataport provides the following capabilities:
Exposes LAS data as a table with fields set up according to the LAS record type used in the file.
Automatically works with LAZ files, which are a compressed version of LAS files.
Analyzes the layout of LAZ files and caches uncompressed data accordingly. This significantly improves performance.
Parses coordinate system info within LAS files, and automatically assigns the specified coordinate system.
If the LAS files are accompanied by any PRJ or PAM XML files, the coordinate system those files specify will be used.
Exposes a classification table, whenever available within the data set.
Exposes any extra fields added via variable-length records.
Exports table data as a LAS or LAZ file. The LAS record type is deduced automatically from fields.
We can link a file using either File - Link or File - Create - New Data Source. If we do not need any extra options offered by the New Data Source dialog, we simply use the File - Link dialog.
To link a LAS or LAZ format file:
Choose File-Link from the main menu.
In the Link dialog browse to the folder containing data of interest.
Ensure the Save cache box is checked (the default).
Double-click the file ending in .las or .laz for the data of interest.
A new data source will be created.
The first time we open the new data source, the opening process will be very slow, while a spatial index is built (a one-time job).
Opening the data source thereafter will be fast, as long as the accessory .mapindexp file accompanies the .las or .laz file.
In Windows Explorer, we have a LiDAR file called pentagon.laz. A LAZ format file is a highly compressed form of a LiDAR LAS file.
In Manifold, we choose File - Link and we navigate to the desired file. The Save cache box is checked by default. We leave it checked to ensure that a .mapcache file is built, to save formatting and other useful info for the next time that we want to link the file.
We double-click on the pentagon.laz file to link it.
A new data source, called pentagon from the stem of the .laz file, appears in the Project pane. We click the + icon to expand the data source and to look into it, to see the drawing and table within. The first time we open the drawing in the data source, Manifold will build a .mapindexp spatial index file. That takes time, a minute or so to build the index, during which the system will not respond and we may see a Windows busy or "not responding" indication. When the index is built, the system will be responsive again.
It takes time to build the index, but that is a one-time job that happens only the first time the .laz or .las file is linked into a Manifold project. Once the index is built, it is saved in an accessory .mapindexp file with the .laz or .las file, and the next time any Manifold session either opens this project or links that .laz or .las file, the existing .mapindexp file will be used and the open will be fast, with no need to build the index again.
We double-click drawings to open them. For a more interesting display, we first create a new data source using a Bing street maps image server as shown in the Example: An Imageserver Tutorial topic. We then create a map and drag and drop the Bing layer into the map, and then we drag and drop the pentagon Drawing into the map.
The drawing appears in default formatting as a mass of gray points. It is located in the right place since LAS/LAZ format almost always provides projection information.
What we see should be regarded as a summary of the data set, a representative preview. This particular data set contains over eight million points, but only a few thousand are shown as a representative sample. LiDAR data sets can contain billions of points, so when they are linked and Manifold runs a spatial query to fetch data for display, the data structure automatically thins the points in the query area and returns a limited number of the best representatives. The number of points displayed depends on the display window size.
If we open the drawing's table, we see it is a read-only table: linked LAS data sets are read-only.
Switching to the Layers pane, we can see the many attributes each LiDAR point has in this data set. One reason Manifold provides a specialized, high-speed, spatial index for points in LiDAR data is to enable more efficient queries against large LiDAR data sets, with possibly billions of points.
After import, if we use Windows Explorer to look at the folder from which we linked the pentagon.laz file, we see two new files have been created, a pentagon.laz.mapcache file and a pentagon.laz.mapindexp file.
The pentagon.laz.mapcache file is the standard .mapcache accessory file Manifold uses to facilitate work with linked files. It is shown above with zero size because we have not yet saved the project. Until we save the project, the .mapcache information is kept cached in memory.
The .mapindexp file contains a specialized spatial index for points that is created when linking LAS LiDAR point cloud files into a Manifold project. The .mapindexp file is then used to service spatial queries, including internal spatial queries such as those performed to display the point cloud within Manifold. The file is versioned so that future Manifold builds can choose to reuse files created by an earlier build or overwrite them with new versions. The .mapindexp file is only created for a linked LAS/LAZ file in a writable location.
The size of a .mapindexp file is significantly smaller than the size of an uncompressed LAS file on which it is built, but it is usually larger than the size of a compressed LAZ. In the example above, the .mapindexp file is over three times the size of the .laz file.
Even though the data source for the linked .laz file is read-only, we can still format the drawing, with Manifold automatically storing the formatting in the associated .mapcache file.
We use a thematic format for point fill color as seen above, using a square point symbol with a stroke size of .5 point and a symbol size of 8 points. We use the Color Brewer CB Spectral palette in reverse order.
Scanning data to acquire statistics when we press the Tally button typically takes 40 to 50 seconds on a data set this size, about 8 million points. Once the data set is scanned, they will be saved while the window is open with no need to recalculate statistics when trying out different intervals for thematic formatting.
Using the thematic format provides a somewhat better visual preview of the data set. The display shows LiDAR elevations for the Pentagon in the United States.
When images are re-styled, the visual results for the same palette may be slightly different as different sets of representative points are sampled by the data structure, based on the size of the viewing window.
When we zoom further into the data set, more points will appear, to provide a more detailed view as we zoom in.
More points will fill in as we zoom further into the data set, to the full resolution of the data set.
When we save the project the .mapcache file will be saved with full information to speed up access to the linked file in the future, either from this project or from any other Manifold project that will link to the same .laz file.
When we save the project, it will be very small, only one MB, since all of the LiDAR data remains stored in the .laz file and the spatial point cloud index data is stored in the .mapindexp file.
Viewing LiDAR as colored points in a vector drawing is neither efficient nor appealing, at least not when points are thinned to representative samples.. We can do better by interpolating the vector point layer into a raster image using the Kriging transform template, as discussed in the Example: Vector to Raster using Kriging topic.
With the focus on the pentagon Drawing layer in the map, we switch to the Transform pane and choose the Interpolate, Kriging template. The LiDAR data is fairly detailed so a Resolution of 1 meter seems appropriate. If we would like to know more precisely, we can zoom far into the data to where individual points start spreading apart, and then use the Tracker tool to measure the approximate spacing between LiDAR points, to see if the spacing between points is approximately one meter.
The pentagon.laz file saves LiDAR data using a meter-based State Plane coordinate system, so that is the coordinate system used by the drawing LiDAR files sometimes are published using degee-based coordinate systems such as Latitude / Longitude, in which case the Resolution and other parameters for Kriging transforms will use units of degrees or radians. That can require some quick use of the Windows Calculator accessory to calculate a sensible value in fractions of degrees or radians to use for Resolution, given the latitude of the data set.
We press Add Component. Manifold calculates for a minute or two, and the result is a new image in our Project pane.
We and drag and drop the new Kriging image into the map. In the illustration above, we have double-clicked the drawing layer to turn it off.
The new image appears all white, for lack of contrast, until we use the Style pane to apply interpolated grayscale color:
We use the settings above, two intervals with interpolation from black to white, to apply grayscale color based on the single channel, the height, in the image.
The result is a grayscale image where higher heights are brighter. That is an interesting effect, but we can do better by applying a colorful palette.
We apply a color palette, the CB Spectral palette in reverse order as seen above. Press Update Style.
That provides a more interesting display. We can make it more interesting by adding shading.
Switching to the Options tab in the Style pane, we check the Use shading box, use a Z scale of 0.003. We press Update Style.
The use of shading provides a nice, 3D effect.
We can zoom further into the image to see the effect that a one meter resolution image generated from detailed LiDAR data can provide. The flat rectangle in the lower right of the window is the Metro-Pentagon station in the lower right corner.
So far, this topic has shown how to link a LiDAR data file into a Manifold project. Linking takes advantage of the specialized .mapindexp point index Manifold generates for point data as found in LiDAR files.
We can, if we like, import a LAS file into a Manifold project, fetching the data from the file and bringing it into the Manifold project. Importing the data into a Manifold file has the plus of very durable .map format, but it does mean that when we save the project the resulting .map file will likely be hundreds of megabytes or a few gigabytes in size.
To import from LAS or LAZ format:
Choose File-Import from the main menu.
In the Import dialog browse to the folder containing data of interest.
Double-click the file ending in .las or .laz for the data of interest.
A table and a drawing will be created.
Using the same pentagon.laz file as in the linking example, we choose File - Import and import the file.
As before, we create a new data source using a Bing street maps image server as shown in the Example: An Imageserver Tutorial topic. We then create a map and drag and drop the Bing layer into the map, and then we drag and drop the pentagon Drawing into the map.
When importing LiDAR data, current Manifold builds create an rtree spatial index on the data and, unlike the data structure used for linking, do not thin any points. The entire data set is shown.
Opening the drawing's table, we see it is fully read/write, since the data is fully resident in the .map project.
Switching to the Layers pane we see all attributes have been imported as well.
As with the linked example, we can use the Style pane to assign a palette to color the elevation values in the drawing.
We use the Style pane to color the points using a palette, but slightly differently than in the linked example since we have a denser set of points. We assign the same palette to coloring both the stroke color and fill color for points. We have used the CB Spectral palette in reversed order.
That provides a more sensible display than in the linked example, where point thinning lowered the apparent resolution.
Zooming in, we can see better detail.
Even though we have all points displayed when importing LiDAR data, we can generate a better visual impression by using Kriging to create a smooth, raster surface. It takes only a minute and a half to create a raster with Kriging, repeating the same steps used for Kriging interpolation in the linked file example.
We drag and drop the resulting image into our map. It appears all white, for lack of contrast, until we use the Style pane to apply interpolated grayscale color:
We use the settings above, two intervals with interpolation from black to white, to apply grayscale color based on the single channel, the height, in the image.
The result is a grayscale image where higher heights are brighter.
We can apply a color palette, the CB Spectral palette in reverse order, plus hill shading, as seen above.
The image again appears very detailed because we have used Kriging to create a raster surface with a pixel at every meter. The result is identical to the linked example, since the data is identical, the only difference being whether it is stored in the original LiDAR .laz file or stored, as a result of import, within the .map file.
A more detailed look at the same image is in the Gallery page of the Manifold website.
We can export point data to LAS or LAZ format by opening the table containing LiDAR data and choosing File - Export, picking LAS Files (*.las, *.laz) as the file type. The table must have the fields required by the LAS standard. If we do not specify a extension for the file name in the Export dialog, a compressed LAZ format file will be created. If we specify a .las extension, an uncompressed LAS file will be created.
Visualization - Manifold LiDAR features are evolving rapidly and are subject to change. Check out the latest additions in the Changes and Additions topic. Current facilities are aimed at analyzing and manipulating data, for example, combining data sets or creating subsets of data, generating raster representations of the point cloud using facilities such as Kriging interpolation and so on. The focus (for now) is not on automatic generation of visual renderings in 2D or 3D. Future builds are expected to add such visualization.
Projections (Coordinate Systems) - Manifold will get projection (coordinate system) information from any PRJ or PAM XML files that accompany the LAS or LAZ files if the accompanying files specify a different coordinate system than the LAS file specifies, the accompanying file wins, since people at times use accompanying PRJ or PAM XML files to fix wrong coordinate information specified within the LAS.
Thank you, Martin! - LAZ file format uses lossless compression, implemented in the open source LASzip compression library, developed by Martin Isenburg, the creator of LAStools. Manifold uses the LASzip library to read/write LAZ files. That helps preserve use of a single LAS / LAZ file format standard, without forking proprietary variations, like ESRI zLAS.
LAS and LAZ speeds compared - Consider an example using a LiDAR data set for part of the city of Portland, Maine, in the US. In uncompressed LAS format, the file is 458 MB. In compressed LAZ format, the file is only 58 MB. To compare the speed of LAS files with LAZ files, in Manifold we used the librarylas dataport to create a data source on a folder containing the LAS file, and then we created a data source on a folder containing the LAZ file. We dragged and dropped both resulting drawings into a map, with timings shown below in a Log Window report.
The Render lines depend upon how fast the data can be fetched from the source. In the case of the LAS file, no decompression is necessary so rendering happens in a fraction of a second. In the case of the LAZ file, that must first be decompressed into a working, LAS version in memory, so rendering takes over 14 seconds, many times longer. Most users will use cache (the default) for the data source, so the extra time rendering is a one-time cost for only the first time the drawing is used in that particular Manifold session. But when working with ensembles of LAS/LAZ files that may contain dozens or over a hundred files, the extra time to decompress LAZ at the beginning can add up. If we leave data in LAZ form and just link to it, then every time we open a project with those drawings, the first time we open the drawing we will have to wait for decompression into working format in memory before the LiDAR points appear in the drawing.
In both cases in the above example, we had previously linked the files so Manifold has created a 52 MB spatial index file for each. That provides as close as possible a comparison of the time cost of LASzip decompression between using the LAZ file and the LAS file. If this is our first time using a LAS or LAZ file in Manifold building a spatial index will go faster with a LAS file than with a LAZ file, since there will not be additional time required to decompress the LAZ file into working form in memory.
When linking LAZ files into a project, leave the Save cache box checked in the Link dialog. When creating data sources using LAZ files, leave the Cache data checkbox checked in the New Data Source dialog. Those options will ensure cache is used for faster access to working data once the LAZ file has been decompressed behind the scenes. To reduce the size of .map files, we can uncheck the Save cached data between sessions box in the New Data Source dialog. Leaving that box checked will save the cache from each session into the .map file, resulting in a larger .map file, which somewhat defeats the idea of keeping file sizes small by leaving data in compressed LAZ file format instead of decompressing it into larger LAS file format.
A good beginning - LiDAR point cloud capability in Manifold is a new feature that is just getting started. LiDAR users are encouraged to send in suggestions for any changes or additions desired. Stay tuned for many new features and improvements!
Edit - Schema
GRD, ESRI .ASC, .GRD
GRD, Surfer .GRD
Example: Spectacular Images and Data from Web Servers
Example: An Imageserver Tutorial
Example: Vector to Raster using Kriging