KML, KMZ Google

KML is an XML-based format with a three-letter file name extension of .kml originally used for annotations in Google Earth displays.  KMZ is exactly the same format compressed using "zip" compression with a three-letter file name extension of .kmz.    Manifold automatically decompresses KMZ to get the resulting KML.   In this documentation we use KML as a term to mean either KML or KMZ.   KML can be used for points, lines or areas with typically two text attributes, a Name field and a Description field, per object.   Manifold can connect to KML files on our computer or, using the Web Server: kmlserver choice in the New Data Source dialog, Manifold can connect to a KML file hosted by a web server.  See the KML Servers topic for connections to KML files on web servers.

 

Caution:  KML is a dangerous format for privacy, because KML files can contain links that trigger downloads from external web servers.  KML files that contain links to other files are called recursive KML files.  Triggering such links can allow third parties to track your activity, to discover the IP address of your computer, and may allow artificial intelligences to determine your identity and to track your activities online.   Google loves that stuff, but you might not.  

 

 

 

Importing a KML

A quick way to import a non-recursive .kml file into a project is to use File - Import, and a quick way to link to a non-recursive .kml file is to use File - Link.   

 

For better control over the resulting import, many Manifold users will import a .kml using File - Create - New Data Source, since the New Data Source dialog provides a Merge layers option, as well as automatically following links within recursive KML files to one file deep by default.

 

To import from kml format:

 

  1. Choose File - Create - New Data Source from the main menu.

  2. In the dropdown menu, choose More... to launch the New Data Source dialog.

  3. Choose File: kml as the Type.

  4. In the Source box, enter the URL to the file, use the [...] browse button to find and to open the .kml file desired.

  5. Check the Merge layers box if desired.

  6. Increase Link depth to follow links more than one additional file deep.

  7. Press Create Data Source.

  8. Expand the new data source hierarchy, highlight the linked components within, and choose Copy.

  9. Click into a blank part of the project outside of the data source hierarchy and choose Paste.

 

Shortcut:  Instead of copying and pasting from within the new data source into the local part of the project, we can just drag and drop components into the local part of the project.  That has the same effect as Copy and Paste when dragging and dropping from a read-only data source into the local part of the project.

 

.kml drawings will often include "folders," that is, layers.  Each layer will be imported into Manifold as a separate drawing so that they can appear within a map as layers. A map component containing all the layers will also be created.   Some .kml files may contain a bewildering number of layers that are best organized within a single drawing.   Checking the Merge layers option box in the New Data Source dialog will create a single drawing that contains all such layers

New Data Source Dialog and Controls

In the main menu, choose File - Create - New Data Source.   The dropdown menu provides a list of favorites to choose from as well as a More... option.  

 

 

Choose More... to launch the New Data Source dialog.   Choose File: kml in the Type box.

 

 

Name

Name for the new data source, "Data Source" by default.  Specify a more  memorable name as desired.

Type

Choose File: kml to connect to a .kml or a .kmz file.

Source

The path to the .kml file.  Use the [...] browse button to navigate to, and to open the desired .kml file.

 Browse button

 

Click to populate the Source box.  Browse the Windows file system to locate and open the desired file using a Select File dialog modeled on the standard Windows File - Open dialog.  The type of file will be pre-loaded into the dialog depending on what was selected in the Type box

Open as read-only

Open the data source read-only.  Not checked by default so that if we want to open a .kml file read/write we can do so.   .kml format will lock out other users if we connect to a .kml file read/write.  Therefore, if we would like to allow other users to be able to simultaneously connect to this same .kml file and we do not need to write to it, we can check this box.  We can also check the read-only box if we would like to protect against accidental changes to the  .kml file

Cache data

Create and utilize an accessory .MAPCACHE file for improved performance. Checked by default.

Save cached data between sessions

Save the cached data for the next time this project is opened, either within the .MAPCACHE file or within the .map project itself within a Cache subfolder within the System Data hierarchy.

 

Checked by default, as it is for most file storage.  KML files tend not to store very large data so there is normally little overhead to caching them within the project, but that helps performance given how slow KML/KMZ is as a format.

Cache only data that is expensive to compute dynamically

If there is significant overhead in computing data to be displayed, cache the results.  Checked by default.

Merge layers

Bring data from all folders within the .kml into a single drawing and table. Add a field for each record in the table giving the name of the original folder.   Most Manifold users will check this option in the case of KML files that contain many folders, since it is usually easier to manipulate data within a single drawing in tasks such as styling the data than it is to work with a large number of layers (often the case with .kml files).

Link depth

The depth of links for recursion.  Default is 1 , which means to follow any links to just one level deep.   To disallow following any links, set this to a value of 0.

 

Set the Link depth setting to greater numbers to follow links to more levels.  For example, set Link Depth to 2 to follow links two levels deep, as in the case of a KML file that contains a URL to a KML on a web server, which in turn contains a link to a JPEG file on a different web server.

 

Caution: The default setting of 1 allows the KML file to launch an external link one level deep.   This is great for convenience, but it may be a risk to privacy.

Create Data Source

Create the new data source in the project pane and close the dialog.

Edit Query

Launch the Command Window loaded with a query that creates the data source using the given settings.  A great way to learn how to use SQL to create data sources.

Cancel

 Exit the dialog without doing anything.

KML Files that Launch External Links

KML files may contain external links, like URL links, that point to external KML files that contain the actual data.  The idea is that instead of a large KML file that contains data, a small KML file can be published and downloaded that simply contains a URL to a big KML that is on some server.   When a user downloads that small KML file, whatever software opens the KML can read the URL, make a connection to the server, and use the KML it finds on the server to download the desired data.  Such KML files are also known as recursive KML files, because one KML file can call another KML file in turn.  

 

That mechanism also makes it easy for the publisher of the KML to ensure the most recent data is always available:  only the KML file on the web server need be updated whenever the data changes.   Anyone opening a KML downloaded earlier will automatically get the latest version by following the URL to the latest version.  While convenient, that mechanism poses a significant privacy risk, as discussed in the Notes section below.  

 

 Caution: Nothing about the name or three-letter extension of a KML or KMZ file warns that it is not just local content, but may launch a connection to an external server.    If we have not looked inside the text contents of a KML to see if it contains a URL, it is safest to always assume it might launch a connection to an external server.

 

Manifold tries to strike a balance between supporting the great convenience of recursive KML files, and protecting inexperienced users from files that connect to alien computers with no warning at all.   That balance is struck with different default behavior between the New Data Source dialog, and the File - Import and File - LInk dialogs:

 

 

Example

We have downloaded a file called USGS Magnitude 2.5+ Earthquakes, Past Month.kml from a USGS server.    We may think the file contains a list of locations of earthquakes, but if we open it in Notepad  (KML files are just text files, which we can open with Notepad), we see it contains:

 

<?xml version="1.0" encoding="UTF-8"?>

<kml xmlns="http://www.opengis.net/kml/2.2" xmlns:gx="http://www.google.com/kml/ext/2.2" xmlns:kml="http://www.opengis.net/kml/2.2" xmlns:atom="http://www.w3.org/2005/Atom">

<NetworkLink>

         <name>USGS Magnitude 2.5+ Earthquakes, Past Month</name>

        <open>1</open>

        <Snippet maxLines="1">Updates every 1 minutes</Snippet>

        <Link>

                <href>https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_month_depth.kml</href>

                <refreshMode>onInterval</refreshMode>

                <refreshInterval>60</refreshInterval>

        </Link>

</NetworkLink>

</kml>

 

The KML contains no locations for earthquakes.  Instead, it contains a link to a file on a USGS server that contains data on recent earthquakes:

 

https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_month_depth.kml

 

The KML file expects whatever software package opens it to connect to the earthquake.usgs.gov server and to grab data from the 2.5_month_depth.kml file on that server.   That server will, of course, log the IP address of our computer when Manifold or any other package connects, but if we trust USGS not to track our activity we probably will not care about that.  

 

If we try to import that KML file with File - Import, nothing will happen and no drawings will be created.    If we try to link that KML file with File - Link, a new linked data source will be created, but when we open it, it will be empty.  In both cases, no drawings appear because the dialogs will not follow links to external servers.   To do that, we use the New Data Source dialog.

 

 

We choose File - Create - New Data Source and then in the New Data Source dialog we provide a memorable name for the data source, Recent Earthquakes.  We choose File: kml for the Type, and then we click the [...] browse button to navigate to the KML file we want to use.

 

 

 

In the Select File dialog, we navigate to the file we want, and we double-click it to choose it.

 

 

Back in the New Data Source dialog, we click Create Data Source.

 

 

A new data source appears in the project.  When we expand it, the system takes a moment to follow the link inside the KML to the USGS server and then fetches data from the KML file on that server that was referenced by the URL.   Depending on the speed of our Internet connection (usually fast in modern times) and the speed of the server (could be slow, even in modern times) that might go quickly or it might go slowly.  

 

Eventually, the drawings synthesized from the KML on the server appear within the data source in the project.  Manifold will cache them (we thoughtfully left the cache options turned on in the New Data Source dialog) so performance can be faster once the data is fetched from the USGS server.

 

 

Manifold's KML dataport creates a map, called USGS Magnitude 2.5+ Earthquakes, Past Month Map,  that hosts all the layers extracted from the KML.  However, like the layers that map is created within the KML data source, so it is read-only.   

 

To create a convenient copy of that map that we can manipulate, for example, adding a background Bing layer or changing the styles used for the drawings, we drag and drop the USGS Magnitude 2.5+ Earthquakes, Past Month Map from within the data source to the main part of the project, below the System Data folder.  

 

Dragging and dropping it does not move anything, since the KML data source is read only, but it does conveniently create a copy of USGS Magnitude 2.5+ Earthquakes, Past Month Map that has links backwards into the data source.  That is just a fast way of creating a map with layers from the data source.  Creating a copy of the map using drag and drop is faster than first creating a new map and then dragging and dropping the various KML layers into that new map.

 

We also create a Bing streets image server data source, to use as a background layer in the new map.

 

 

The above illustration shows our new map after we have added a Bing streets layer.   We have styled the various drawing layers in the map and we have used the Layers pane to group them into a folder.   We have also grouped the many labels layers into a folder, making it easy to turn all the labels layers on and off with a single clic.

GroundOverlay

KML files can contain GroundOverlay references to images that are stored in other files and which are intended to be used as part of the KML.  Using the New Data Source dialog, Manifold can automatically import GroundOverlay images that are stored on the local machine or which are accessible through local network references as part of the KML import.  

 

 

 

See the Example: Import KML with GroundOverlay Image topic for an example and discussion.

Example

We will import a .kmz file that provides historic data for tracks of hurricanes in the Atlantic ocean.  Launch File - Create - New Data Source.  

 

 

Choose More... to launch the New Data Source dialog.

 

 

Enter hurricanes as the name and for the Type choose File: kml.  No need to cache, so we uncheck the sub-boxes under Cache data.   

 

This particular .kml uses a spectacularly messy set of folders, one for each hurricane, so we will check the Merge layers box to consolidate those many folders into a single drawing.  Click the [...] browse button

 

 

In the Select File dialog, navigate to the desired .kml file, click on it, and choose Open.

 

 

Press Create Data Source.

 

 

 

In the Project pane we expand the new hurricanes data source and we ctrl-click on the four components within to highlight them.   We press Ctrl-C to copy them.   We click into an empty part of the Project pane outside of the data source hierarchy and press Ctrl-V to paste.    We can now click on the hurricanes data source and choose Delete to delete it.

 

 

What we have just done is to link to the .kml file and to then copy the data in it and paste that into the local .map project.   That is the same as using File - Import, except that we accomplished the import using the New Data Source dialog that gave us the option of merging many folders within the .kml into a single drawing that is much easier to work with.

 

We double-click the Map component to open it.

 

 

The result is a typical KML drawing, with a labels layer above, that is brought in from a .kml.   We will remove the labels layer from the map since this particular KML does not use labels.

 

 

We color the drawing layer with lighter colors using Style as seen above.   The Folder field in the table specifies the name of the layer from which a particular object was imported.  We can use that in a thematic format to color each object by the layer from which it originated, which usually results in a comprehensible display.   We have applied the CB Paired palette, using a method of unique values.

 

 

The result is a much more comprehensible display, seen above a Google satellite web server layer in a map.

Without Merge Layers

Using the Merge layers command makes a big difference.

 

 

Simply importing the same .kmz file without using Merge layers results in each track having its own folder.

 

 

The map that is created includes the labels component and the drawing within each folder as layers, resulting in many layers, each one of which must be styled individually.   

 

 

In the map illustration above we have used the Layers pane to apply partial opacity to the Google background layer to make it lighter, so the default black color of the hurricane tracks is more visible.  Even using the Layers pane to organize layers, having dozens of layers in a map with one layer for each line is extremely tedious.  It is far easier to use Merge layers to create a single layer.

Projections

KML and KMZ always provide data in Latitude / Longitude projection using WGS84 datum.   Manifold will automatically assign Latitude / Longitude coordinate system to drawings imported from KML.

Multi-type Geometry

KML, like GML, GeoJSON, and similar formats, can include geometry data for each object that mixes one or more geometry types.  Geometry values declared as multi-type (In KML, using the tag <MultiGeometry>) but with all parts having the same underlying type, such as all areas, all lines, or all points, will preserve that underlying type. Geometry values declared as multi-type that mix different types are converted as follows:

 

 

For example, if a KML file contains multi-type geometry with the same object defined both as a point and as a line, such as:

 

         <MultiGeometry>

            <Point>

               <coordinates>134.05796957910275,45.56434225030646</coordinates>

            </Point>

            <LineString>

               <coordinates>135.13128666,48.42547611 134.99108892,46.64727784 131.96051024,43.13012695</coordinates>

            </LineString>

         </MultiGeometry>

 

The above object will be read as a line.

 

 

The use of multi-type geometry is surprisingly popular, an example being the above data set, imported from KML, showing world pipelines downloaded from the worldmap.harvard.edu site.

 

Notes

Selection and Viewing Attributes in the Drawing - As imported, the drawing does not have any index so neither selection nor alt-clicking on an object in the drawing will work.   A drawing's table must have a key field with an index in it for selection or for Alt-click or Shift-Alt-click to work in the drawing.  If the table does not have an index, choosing an object by alt-clicking will not be available.  To add an index to the table, see the Add an Index to a Table topic.

 

Privacy risks from KML Files that contain links - KML files that contain links can be a risk for privacy.  Nothing about the name or extension of a KML file that contains links warns users that it will open a connection across the Internet to some other server.   

 

Users might think the KML is a local file, like a shapefile or a GeoTIFF, without realizing that using the KML will issue a command to connect to a server under someone else's control.  That may not be a big deal if the people who publish the KML or who operate the servers those files contact are friendly, but it can be a very big deal if they are malevolent.

 

Suppose, for example, Jon is a peace activist who uses GIS to uncover secret military bases of repressive regimes.  He uses a VPN with his browser when he browses the web.  One day he finds a web site that publishes a KML that lists the locations of secret military bases operated by a repressive regime.  Jon downloads the KML file through a VPN, so no one can tell the IP address of the computer that downloaded the file.  

 

But the KML Jon downloaded does not contain the actual list of secret bases and their geographic coordinates. Instead, it contains a URL link to a KML file that is located on a web server.  When Jon opens the KML file in his GIS package, if that GIS package supports the KML standard's use of links, it will connect to the web server based on the URL within the KML file. fetching the contents and displaying them for Jon.

 

When Jon sees the contents appear, he thinks he just opened a local KML file, without realizing that he has connected through the web to a server that has recorded the real IP address of Jon's computer.   If the "secret bases" KML file was published as phishing bait by a repressive regime, that regime now also knows the IP address of Jon's machine.  That could be a big problem for Jon if he lives within the jurisdiction of that repressive regime, or if that regime can get to him through extradition or covert action.  

 

That is a malevolent scenario, but privacy risks arise in more common situations, where KML files may contain small URL links that do nothing but serve as a way for companies to know what people are interested in, based on the KML files they download.  Such information helps those companies build profiles and to connect online activity with real identities.   If we have clicked open a KML file that uses a link to get locations of clothing-optional beaches, and then suddenly Google starts populating our browser sessions with ads for nudist resorts, we know that may have happened.  

 

See Also

Style

 

Web Servers

 

KML Servers

 

Example: Import KML with GroundOverlay Image - files can contain GroundOverlay references to images that are stored in other files and which are intended to be used as part of the KML.  Manifold will automatically import GroundOverlay images that are stored on the local machine or which are accessible through local network references as part of the KML import.