Web Servers

 

Web Servers and Image Servers

il_wikimapia_munich_med.png

il_colosseum_sm.png

Manifold has the ability to bring data into a project from a seemingly endless range of different types of data sources, including almost any file format or database management system.

 

In addition, Manifold also has the ability to reach out through Internet to fetch spatial data, both vector drawings and raster images, from web servers that provide such data using a variety of well-known protocols.   

 

Such web servers typically provide data dynamically to fill in whatever view we are looking at in a window that shows content from the web server.  As we pan or zoom in or out in the view Manifold automatically tells the web server what view we are looking at and the web server responds by sending the data needed to fill in the view.  Manifold begins by fetching tiles closer to the center of the requested area whenever possible, to produce a usable picture faster.  We normally do not need to wait for the entire screen to fill before zooming in to what we can already see.

 

This topic discusses web servers and image servers.  Image servers are a very popular form of web servers which use their own protocols and have become so famous, such as Google Maps, that they are worth supporting with pre-configured modules.  Image servers are often referred to as imageservers.   The illustrations in this topic show examples of different web servers and image servers.

il_lidar_md_sm.png

 

To create an image server data source:

 

  1. Launch Manifold.

  2. Choose File - Create - New Data Source, or use the Project pane context menu.

  3. In the Type box choose Web Server: imageserver.

  4. In the box just below the Type box choose the desired image server, for example, Bing Maps Street Map.

  5. Press OK.

  6. The new data source appears in the project.  Click on the + icon to expand it.

  7. Double-click on the image within to open it and see what the image server provides.

 

See File - Create - New Data Source for instructions on connecting to many different types of web servers, including custom web server connection strings.

 

For examples, see the Example: An Imageserver Tutorial topic or the Example: Create a New Data Source from a Manifold Image Server topic.

Favorites

Manifold comes pre-loaded with several image servers in the Favorites for data sources, typically Bing streets, Bing satellite, Google streets, Google satellite and OpenStreetMap streets.  To use one of those we can choose File - Create - New Favorite Data Source. and simply click on the image server desired from the list of favorites that appears.

 

il_bing_europe_sm.png

To create a web server data source:

 

  1. Launch Manifold.

  2. Choose File - Create - New Data Source, or use the Project pane context menu.

  3. In the Type box choose the desired web server technology, for example, Web Server: wms.

  4. In the Source box enter the connection string / URL required for the desired server.

  5. Press OK.

  6. The new data source appears in the project.  Click on the + icon to expand it.

  7. Double-click on the image within to open it and see what the we server provides.

 

See File - Create - New Data Source for instructions on connecting to many different types of web servers, including custom web server connection strings.

 

For an example, see the Example: Style Applied to an Image Server Image topic.

Web Server Data Sources

Web servers normally send either raster data such as images or they can send vector data that represent objects in drawings.  When they serve raster data such data normally is sent in the form of tiles, rectangular groupings of pixels that are 128 x 128 pixels or 256 x 256 pixels or some other convenient size.   Sending tiles through the web is much more efficient either than sending individual pixels or computing on the fly and sending a screen's worth of pixels.   For that reason such web servers are often called tile servers.

il_esri_physical_sm.png

Besides the obvious use of pixels and imagery in the form of tiles within web servers that provide imagery such as overhead satellite photos, as, for example, Google or Bing Satellite view do, web servers will often also use imagery to provide a visual representation of fundamentally vector data as well.   

 

For example, street maps that show roads are maintained on the servers as drawings, that is, vector data.   To show those street maps on the web, the server creates an image, like a screen shot, of the vector data complete with all formatting, labels and other accessory graphics arts wonderfulness that helps create an appealing map.   The server then sends that imagery in the form of tiles out over the web.  

 

Sending a visual representation of vector data out over the web in the form of imagery, that is, screenshots of the data, is a good thing for people who just want to look at the map.  They can look at the map using ordinary web browsers or simple software on their telephones, tablets or computers that need not do anything more complicated than put together images from tiles that consist of well-known, simple image formats like jpg or png.   

 

But that is not a good thing for people who want access to the actual vector data so they can use it for GIS purposes.   For example, if we want the actual vector line objects that make up the boundaries of real estate parcels so we can accurately calculate the areas inside an image where the boundary lines are just fuzzy swaths of different-colored pixels (as they are in an image) is not the data we need.    For such purposes there are protocols, such as WFS, that can be used by web servers to provide the actual vector data served dynamically.il_sphinx_sm.png

Image Servers

Manifold Image Servers, often written in concatenated form as imageservers, are a subset of web servers that serve imagery in a form supported by dedicated modules specific to each such image server.   Such image servers typically do not utilize an "open" protocol but use their own specific protocols and methods to do something similar, serving tiles dynamically on demand to fill in a desired view.

 

Manifold image server modules are snippets of code which use the openly published Manifold Image Server Interface (ISI) to automatically fetch image tiles from popular image servers like Bing, Virtual Earth, Wikimapia, Yahoo!, Google Maps, Yandex and many others. Image servers can provide street maps, overhead satellite imagery, combinations of streets and satellite imagery and other data as well.   

 

Anybody can write an image server module using the ISI interface and many such modules have been written by users and published for others to use.  

 

Many image server modules are already built into Manifold.   Installing an additional, user-written image server module is easy: simply place the DLL for the image server module within the Manifold installation path, typically in the ~/Shared/Bin folder.   When Manifold or some other Manifold product that supports ISI launches it will automatically utilize that DLL to include the provided image server module.il_eiffel_sm.png

Connecting to Web Servers

Just like we must know the URL of a web site to connect to it, we usually must know the protocol utilized and the URL to the web server which provides the data we would like to use.

 

Some types of web servers, for example well-known image servers such as Google maps or Bing maps, are built into Manifold in the sense that their default connection strings are pre-loaded into the dialogs that invoke them.   Adding dynamic imagery from such servers is as simple as choosing the desired image server from a list in a dialog.    Other types of web servers, for example those utilizing wms protocol are more general and connect to a vast variety of different servers operated by various web sites, individuals and institutions.  To use those we must note the connection string (a form of URL) required and enter it into the dialog.

 

Some web servers are free to use without any restrictions at all.   Some utilize access controls such as the need to enter a special key or to incorporate login credentials as part of the connection string.   Manifold dialogs assume the user is capable of visiting the desired web site to learn about any restrictions or special credentials required to access a given server.

 

Some web servers, often those which use WFS or similar to provide vector data, can be astonishingly slow.  If no data appears when using such servers, wait for a minute or two and check the Log Window to see what is going on.

 

See File - Create - New Data Source for instructions on connecting to many different types of web servers, including custom web server connection strings.

 

il_b2_med.png

 

Cache

The New Data Source dialog provides the option to Cache data when using a web server source.   

 

il_webserver_cache01_00.png

 

Using cache can provide faster performance with slow web servers because data that is reused, say, because we return to a previously-viewed scene, need not be re-fetched.  If we are no longer connected to Internet or if the remote server goes down, the data source can still show what tiles have been downloaded into cache.    

 

Cache data for image servers is stored in cache tables within the Cache folder in the System Data hierarchy.    If cached data is no longer desired, individual cache tables or even the entire Cache folder can be deleted to reduce the size of the project file.  When the data source created with the cache option enabled is opened again, the Cache folder and cache table for that source will be re-created so the cache can grow again.

 

il_webserver_cache01_03.png  il_webserver_cache01_04.png  il_webserver_cache01_05.png

 

The downside to using cache is that cache tables can accumulate many megabytes of data within the project file.   Beginners should uncheck the Cache data box since it is easy to forget that caching data and then browsing can grow apparently small projects to large size.   See the discussion and illustrations in the File - Create - New Data Source topic.

 

Notes

Problems connecting - Check the Log Window to see what is going on behind the scenes if an attempted connection does not work.     The problem is usually a wrong connection string, failure to provide required credentials such as an key string that authorizes access, wrong choice of protocol (the server uses WMS and the user picks something else), an incredibly slow server, a server that is offline or a server that is wrongly configured and which is not correctly using the protocol it claims to use.  Visit the Manifold community forum and talk out difficulties with other users.    The very last thing to consider is the possibility that a bug in Manifold is causing the problem.  Bugs happen, of course, but leaping to the conclusion as a first step that a bug is the problem is almost always a mistake.tech_ravi_sm.png

 

Web servers are often slow or mis-configured -   Some web servers to which we might connect using WMS, WMTS, WFS or other connection standards can be incredibly unresponsive or, despite allowing a connection, will not allow data to be fetched.   If pressing the Test button results in a successful connection, or if layers available are displayed when we expand the data source, but then when we try to display an advertised layer nothing happens, we can open the Log Window to see what is going on.

 

Here are a few lines of log window information for a happy session, connecting to a WMS server in New Zealand run by the ever-efficient LINZ organization:

 

Web request: (root)::[Data Source] (GetCapabilities) (0.578 sec, 13.0 KB)

Web request: (root)::[Data Source]::[layer-767] (GetMap) (1.255 sec, 47.9 KB)

Render: [Data Source]::[layer-767 Image] (1.561 sec @1.0)

 

In contrast, a few lines of log window information for an unhappy session, attempting to connect to the WMTS server run by the French cartographic service IGN but getting no image for a cadastral parcels layer:

 

Web request: (root)::[IGN]::[CADASTRALPARCELS.PARCELS_bdparcellaire_b] (GetTile) (0.070 sec, 0 B)

*** (root)::[IGN]::[CADASTRALPARCELS.PARCELS_bdparcellaire_b] (GetTile) The remote server returned an error: (403) Forbidden.

Render: [IGN]::[CADASTRALPARCELS.PARCELS_bdparcellaire_b Image] (0.315 sec @1.0)

 

From the above we see an empty cadastral parcels image was rendered because the remote server refused to provide tiles in response to the GetTile request, returning a 403 error Forbidden.    Finding out why a web server refuses service can involve some detective work.  

 

The server might require a special key for access, a key we are trying to use may be out of date, a login and password may be required for some resources, the connection string may have been copied and pasted inaccurately or, of course, in the case of IGN, the server may be refusing to work to show solidarity with striking railroad workers or air traffic controllers.

 

Keep in mind that just because a web server offers tiles via various standards like WMTS or whatever does not mean that it will be fast or even reasonably tolerable.  Some web servers are painfully, incredibly slow and will take many seconds or even minutes per tile.   When images do not appear or fill in with painful slowness a look at the Log Window can show the problem is not our Internet connection or Manifold, but just incredibly slow response from the web server.

 

Some WMS servers are erroneously configured so the server mislabels image tiles as text/xml.  A WMS compliant client will reject such mislabeled tiles, but because webmasters often fail to test their sites using WMS compliant clients such mislabeling errors are frequent.  To handle such errors, Manifold includes a simple defensive measure: When Manifold reads WMS tiles that a server labels as non-image data, the system will distrust the server and try common image formats.  If the tiles are images the system will disregard the server's claim such tiles are text and instead will process them as images. Second-guessing such server errors allows Manifold to interoperate with wrongly-configured servers, albeit at the cost of some processing overhead when correctly working servers return text/xml data.

 

ico_nb_arrow_blue.png  Must Have Internet - It seems obvious that to connect to a web server we must have a live, functioning Internet connection.  What might not be obvious is that sometimes we may have only a partial Internet connection, without realizing we do not have a full Internet connection.  A classic example is using a browser, such as Opera, which may ignore failed IPv6 DNS resolution but work OK using IPv4 DNS resolution.  Without IPv6 DNS resolution we can browse Internet using Opera but we might not be able to connect to Microsoft, Google, or some other image servers.   

 

Manifold's web server connections, just like the latest Google Chrome and Microsoft Edge browsers, require full IPv6 DNS resolution.   That normally happens automatically using whatever IPv6 DNS servers are assigned by our Internet service provider, but at times IPv4 DNS continues to work OK while the IPv6 DNS servers assigned by our ISP do not work.  If we are using Opera, and not Edge or some other browser that uses IPv6 DNS resolution, we might not notice that our ISP-provided IPv6 DNS resolution is not functioning.    We might only notice the problem when trying to use an image server or other web server within Manifold and it does not work.

 

If we cannot get a display from a web server, we can check the Manifold Log Window. If the log reports that a connection cannot be made, in Windows network dialogs for our Internet connection  we can try setting the IPv6 DNS servers to some "known good" DNS servers.   A good choice is using Google's public DNS service, using the following addresses for primary and backup DNS server addresses:

 

2001:4860:4860::8888
2001:4860:4860::8844

 

 

Collections of Web Servers - Check the Product Downloads page for pre-package project files in Release 9 / Viewer .map format that Manifold publishes which contain collections of dozens of popular web servers data sources.

 

Important:  Some web servers, notably TMS servers, use embedded command tokens.  Use Manifold's custom setting to connect to those.   See the File - Create - New Data Source topic for how to do that.

 

Imageserver or Image Server? - Manifold uses two styles, with and without a space character, using the two styles as synonyms without any special meaning attached to one or the other.   For historical reasons going back to prior Manifold releases the term imageserver tends to be used in programmatic contexts.   For discussions in ordinary text the term image server tends to be used.

 

Old versions of WMS - Connecting to a WMS server always uses XY axis ordering based on the assumption that servers using old versions (pre-1.1.1) of the WMS standard ignore axis order for the coordinate system.

 

See Also

File - Create - New Data Source - an absolutely key topic for connecting to many thousands of different web servers.

 

New Favorite Data Source - Choose from a drop down list of Favorite data sources, or choose Edit Favorites to launch the Favorites dialog for data sources.

 

Favorite Data Sources - View and edit the Favorites dialog for data sources, showing various data sources that may be added to the project with a single click from the list provided to the New Favorite Data Source menu.

 

Example: Spectacular Images and Data from Web Servers - A must see topic providing a gallery of views illustrating how Manifold can use web servers such as image servers and other free resources to provide a seemingly endless selection of spectacular background maps, satellite images and GIS data with nearly zero effort.

 

Example: An Imageserver Tutorial - An extensive tutorial showing step by step how to add new data sources that are image servers, how to show them as layers in a map, how to create a new drawing that matches the projection of the map and how to trace over what is seen in an image server layer to create an area object in the drawing.

 

Example: Create a New Data Source from a Manifold Image Server - Manifold image server modules are snippets of code which use the Manifold Image Server Interface (ISI) to automatically fetch image tiles from popular image servers like Virtual Earth, Wikimapia, Yahoo!, Google Maps, Yandex and many others. Image servers can provide street maps, overhead satellite imagery, combinations of streets and satellite imagery and other data as well.  Using Manifold Image Servers is one of the most popular Manifold features.

 

Example: Create a New Data Source from a MAP File - Create a new data source from an existing Manifold .map project file.   This is the classic way to nest projects, one calling another, to create libraries of data and projects.   Access to nested projects has effectively zero performance loss and the links within projects take up effectively zero space so we can create huge constellations of data at our fingertips.

 

Example: Create a Data Source within an Existing Data Source - When a data source is writable, for example, if the data source is a Manifold .map file, we can work within that data source as if it were at the top level of our project.   For example, we can create a new data source that is nested within the existing data source.   This example shows how.

 

Example: Create and Use New Data Source using an MDB Database - This example Illustrates the step-by-step creation of a new data source using an .mdb file database, followed by use of SQL.  Although now deprecated in favor of the more current Access Database Engine formats, .mdb files are ubiquitous in the Microsoft world, one of the more popular file formats in which file databases are encountered.  

 

Example: Modify GPKG Geometry with SQL then Add Drawing - This topic provides a "Hello, World" example that shows a simple, but typical, task involving spatial data.  We will take a country-sized data set in GeoPackage (GPKG) format and change all areas in the data to the boundary lines for those areas and then save those boundary lines as a new table.  We add a spatial index to the table and create a new drawing to visualize the new table.

 

Example: Trace an Area in a Map over an Image Background - In a map with a drawing layer above an image layer (served dynamically by an image server), create an area object in the drawing by tracing over the outlines of something seen in the image layer below

 

Example: Style Applied to an Image Server Image - Because the Edit - Style dialog simply changes the way an image is displayed and not the data, it can operate on read-only data served by various web servers such as WMS REST servers.    In this example we look at every detail of creating a data source using a WMS REST  server and then manipulating the appearance of the display with Style.  We will connect to a WMS server that provides LiDAR data in various forms, including as terrain elevation.