File - Create - New Data Source

 

File - Create - New Data Source

The New Data Source dialog allows us to create a data source within a project.    The data source can be created within the existing .map project or within another, nested data source that already exists within the project.  tech_angus_sm.png

 

Important: New Data Sources are linked, not Imported, into the project.  See the Importing and Linking topic.

 

A new data source links data into our project without importing the data.    The data in that new data source remains stored within whatever is the originating source.  See comments on linking data into a project in the Getting Started topic.  

 

Creating a new data source is a more sophisticated way of linking data.   It uses a more comprehensive dialog than a simple File - Link and so is the way we link data from databases, where optional settings such as login or password for the database might be required.

 

To link data as a new data source:

 

  1. In the main menu choose File - Create - New Data Source.

  2. In the New Data Source dialog provide a Name for the new data source if the default is not desired.

  3. Choose the Type of data source from the long list available.

  4. Choose the sub-type of data source when those are available.

  5. The Source box will be automatically loaded for some data sources, for example, connection URLs to Bing or Google servers.   For other sources, use the [...] browse button to navigate to the file or data source connection as required.

  6. Set option boxes that appear as desired, for example, checking Open as read-only if the data source should be kept read-only.

  7. Press Create Data Source.

 

See the File - Create - New Data Source topic for details.

 

For an example of creating a new data source see the Getting Started topic as well as the many examples recommended at the end of this topic.

 

A new data source can be created from data that is resident in hundreds of different type of data sources.  The New Data Source dialog organizes data sources as four different Types, depending on what we specify to access the data source, to make them easier to find in a long list of available types:

 

 

 

 

 

Connections to the above are provided within Manifold by dataports, which are modules of code within Manifold that provide connection services to different data sources.   Dataports are always at least read-only and will often also provide read/write capability.

 

There may be more than one way to connect to a data source.   For example, we can connect to Microsoft Access MDB file databases directly from a 32-bit Manifold session or we can connect using the generic OLE DB connection.  Normally we would connect directly using the MDB File Database option but if the MDB file has a password for access we can connect using the OLE DB option, which allows specifying a password.

 

Note that we can create a new data source that links a script into the project.  This is a handy way of including within projects scripts that might reside outside of a project, be shared by other programs, edited or updated by other users and so on.   

 

We can also create a new data source, using File: txt that links text files into a project to create comments components.  These are useful when documenting information about other data, sharing "to do" lists and other team-oriented comments and other text documents that might be edited by other users using other programs or created automatically by other processes.

 

Databases

il_newdatasource_oracle.png

 

For important notes when connecting to DBMS data sources, see the DBMS Data Sources - Notes topic.

Controls

Name

Name for the new data source, "Data Source" by default.  Specify a more  memorable name as desired.  If we forget the origin of a data source we can hover the mouse over the data source and a tool tip will provide connection information.

Type

Choose from the list of different types of Database: data sources that may be connected.

 

A generic connection technology such as ADONET will enable a secondary box below the Type box when Database: adonet is selected as the Type.   That secondary box will be loaded with a list of connection options made available by any DLLs within the PATH supporting ADONET connections.  If no such DLLS exist the list will be empty.

Source

A connection string or path to a file or other resource to open.   The source string can be manually entered or it can be created using dialogs launched by the [...] browse button or it may be automatically populated for some types of data sources.

[...] Browse button

Click to populate the Source box.  When choosing a database Type that consists of a generic connection technology such as ODBC or OLEDB the [...] Browse button will launch the Windows dialog for that connection technology, for example, launching the Windows Data Link Properties dialog for OLEDB or the Windows Select Data Source dialog for ODBC.

Open as read-only

Open the data source read-only.  Not checked by default so that if the dataport for the data source supports a read/write connection by default the data source will be both readable and writable.   Some types of data sources, such as enterprise class DBMS servers like Oracle, will allow multiple users to connect read/write to a data source.  Other types of data sources, such as most file formats, will lock out other users if we connect to them read/write.  Therefore, if we would like to allow other users to be able to simultaneously connect to this data source and we do not need to write to it, we can check this box.  We can also check the read-only box if we would like to protect against accidental changes to the data source.

Use vector data as either XY or YX according to coordinate system

This option appears for those data sources where coordinate systems can be in either ordering.   Standards such as EPSG for specifying coordinate systems have codes which sometimes specify that data occurs in XY form, that is longitude, latitude coordinate ordering, and sometimes as YX form, that is, latitude, longitude coordinate order.   In theory, this box should always be checked.  It is not checked by default because, unfortunately, so many popular packages disregard EPSG standards that most spatial data encoded with EPSG codes which require flipped coordinate ordering is not flipped.  See the That YX Thing essay topic.

Create Data Source

Create the new data source in the project pane and close the dialog.

Edit Query

Launch the Command Window loaded with a query that creates the data source using the given settings.  A great way to learn how to use SQL to create data sources.

Cancel

 Exit the dialog without doing anything.

Database Login Dialog

When a Database: data source is the Type, the [...] button next to the Source box launches the Database Login dialog that allows specifying a server and connection characteristics.

 

dlg_database_login_create_nds.png

 

The above dialog provides the following controls:

 

Server

The connection string for the database server.  

Use integrated security

Use Windows integrated security credentials.

Use login and password

Use security credentials specified in the Login and Password boxes, which are enabled when this option is selected.

Database

Specific database to use in this server.  Enabled for database servers such as PostgreSQL and SQL Server that allow specifying a database within the database server.

Test

Press to try a connection to the server using the specified settings.

Specifying ports when connecting to DBMS data sources

When connecting to a DBMS we can specify the port to use either in the connection string or within the name of the server after a colon.   For example, when connecting to a PostgreSQL server we can write for the name of server:

 

localhost:5433

 

The dialog will know we are connecting to PostgreSQL, a database that allows explicit specification of a port to use for the connection, and it will split out the port number from the name automatically.   Alternatively we could explicitly specify the port as part of the connection string, in the form:

 

{ "Source": "host=localhost port=5433 user=postgres password=xxxx dbname=xxxx" }

Files

il_newdatasource_file_manifold.png

Controls

Name

Name for the new data source, "Data Source" by default.  Specify a more  memorable name as desired.  If we forget the origin of a data source we can hover the mouse over the data source and a tool tip will provide connection information.

Type

Choose from the list of different types of File: data sources that may be connected.  .map project files are of type File: manifold.

Source

A connection string or path to the file or other resource to open.   The source string can be manually entered or it can be created using dialogs launched by the [...] browse button.

[...] Browse button

Click to populate the Source box.  Browse the Windows file system to locate and open the desired file using a Select File dialog modeled on the standard Windows File - Open dialog.  The type of file will be pre-loaded into the dialog depending on what was selected in the Type box

Open as read-only

Open the data source read-only.  Not checked by default so that if the dataport for the data source supports a read/write connection by default the data source will be both readable and writable.   Most file formats, will lock out other users if we connect to them read/write.  Therefore, if we would like to allow other users to be able to simultaneously connect to this data source and we do not need to write to it, we can check this box.  We can also check the read-only box if we would like to protect against accidental changes to the data source.

Fault-tolerant save

Appears when File: manifold is selected in the Type box.   When checked (the default) the system will create a .SAVEDATA accessory file and will use fault tolerant saves if a .map file is connected as a new data source.   See comments at the end of the Getting Started topic and in the Accessory Files Created topic.

Cache data

Create and utilize an accessory .MAPCACHE file for improved performance. Checked by default.  This option appears with most file data sources and has two options, both checked by default, one to cache only data that is expensive to dynamically compute and the other to save cached data between sessions.   It only makes sense to uncheck either option if we are desperately low on free disk space.

Save cached data between sessions

Appears as a sub-choice for some data sources for which the Cache data option appears.   Save the cached data for the next time this project is opened, either within the .MAPCACHE file or within the .map project itself within a Cache subfolder within the System Data hierarchy.

Cache only data that is expensive to compute dynamically

Appears as a sub-choice for some data sources for which the Cache data option appears.   If there is significant overhead in computing data to be displayed, cache the results.  For example, some file formats may require complex calculations to extract data from the format.  It is quicker to do that once and save the result for re-use than to repeatedly perform the same computation/extraction.

Create Data Source

Create the new data source in the project pane and close the dialog.

Edit Query

Launch the Command Window loaded with a query that creates the data source using the given settings.  A great way to learn how to use SQL to create data sources.

Cancel

 Exit the dialog without doing anything.

 

Folders

il_newdatasource_folder_library.png

Controls

Name

Name for the new data source, "Data Source" by default.  Specify a more  memorable name as desired.  If we forget the origin of a data source we can hover the mouse over the data source and a tool tip will provide connection information.

Type

Currently supports only the Folder: library type, to support Release 8 image libraries.

Source

The path to the image library folder, along with a template for the file name pattern that includes [X] and [Y] specifiers.   For example, [X]-[Y].* will match image files using names like [01]-[01].png, [01]-[02].png, [02]-[01].png, and so on.

[...] Browse button

Click to populate the Source box.  Often used to launch the File - Open dialog to open a file format or file database format.

Open as read-only

Open the data source read-only.  Not checked by default so that if the dataport for the data source supports a read/write connection by default the data source will be both readable and writable.   Most file formats, will lock out other users if we connect to them read/write.  Therefore, if we would like to allow other users to be able to simultaneously connect to this data source and we do not need to write to it, we can check this box.  We can also check the read-only box if we would like to protect against accidental changes to the data source.

Cache data

Create and utilize an accessory .MAPCACHE file for improved performance. Checked by default.  This option appears with most file data sources and has two options, both checked by default, one to cache only data that is expensive to dynamically compute and the other to save cached data between sessions.   It only makes sense to uncheck either option if we are desperately low on free disk space.

Save cached data between sessions

Appears as a sub-choice for some data sources for which the Cache data option appears.   Save the cached data for the next time this project is opened, either within the .MAPCACHE file or within the .map project itself within a Cache subfolder within the System Data hierarchy.

Cache only data that is expensive to compute dynamically

Appears as a sub-choice for some data sources for which the Cache data option appears.   If there is significant overhead in computing data to be displayed, cache the results.  For example, some file formats may require complex calculations to extract data from the format.  It is quicker to do that once and save the result for re-use than to repeatedly perform the same computation/extraction.

Recurse into subfolders

If checked, recursively scan folders below the location named in Source to find more image files.   If not checked, the dataport will use only image files found in the Source folder.

Reverse Y

The X axis is always numbered in increasing order from left to right.  By default, Y axis ordering is numbered in increasing order from bottom left upwards.   Checking this box specifies Y axis numbering in increasing order from upper left downwards.

Cut margins

The number of pixels to clip from left, right, top and bottom margins. Used when image tiles have borders that should be removed.

Create Data Source

Create the new data source in the project pane and close the dialog.

Edit Query

Launch the Command Window loaded with a query that creates the data source using the given settings.  A great way to learn how to use SQL to create data sources.

Cancel

 Exit the dialog without doing anything.

 

Web Servers

il_newdatasource_webserver_bing_st.png

Controls

Name

Name for the new data source, "Data Source" by default.  Specify a more  memorable name as desired.  If we forget the origin of a data source we can hover the mouse over the data source and a tool tip will provide connection information.

Type

Choose from the list of different types of Web Server: data sources that may be connected.  

 

Choosing Web Server: geocodeserver in the Type box will enable a secondary box below the Type box that will be loaded with a list of built-in geocoding servers, as of this writing: Bing Geocoder, Google Geocoder, Here Geocoder, Here Landmark Geocoder, Here Reverse Geocoder, MapBox Geocoder, MapQuest Geocoder, OpenCage Geocoder, and Yandex Geocoder.

 

Choosing Web Server: imageserver in the Type box will enable a secondary box below the Type box that will be loaded with a list of built-in image server, as of this writing: Bing Maps Hybrid, Bing Maps Satellite, Bing Maps Street Map, Google Maps Satellite, Google Maps Street Map, Google Maps Street Map (Transparent), Google Maps Terrain, Google Maps v2 Hybrid, Google Maps v2 Satellite, Google Maps v2 Street Map, Google Maps v2 Terrain, Here Maps Base, Here Maps Hybrid, Here Maps Satellite, Here Maps Terrain, MapBox Hybrid, MapBox Satellite, MapBox Street Map, MapQuest Hybrid, MapQuest Satellite, MapQuest Street Map, NearMap Street Map, NearMap Street Map (Direct),  OpenStreeMaps Base, OpenStreeMaps Migurski's Terrain, OpenStreeMaps OpenCycleMap, OpenStreeMaps OpenCycleMap Transport, WikiMapia Street Map, WikiMapia Street Map (Transparent), Yandex Maps Satellite, Yandex Street Map, Yandex Maps v2 Hybrid, Yandex  Maps v2 Satellite, Yandex Maps v2 Street Map, Yandex Maps v2 Terrain, Custom.

 

Choosing a built-in geocode server or image server from the list will automatically load the connection string into the Source box.

Source

A connection string or path to the file or other resource to open.   The source string can be manually entered, or it will automatically be populated for Web Server: geocodeserver and Web Server: imageserver types,  or it can be created using dialogs launched by the [...] browse button.

[...] Browse button

Click to populate the Source box.  Clicking the [...] Browse button will launch the Web Login dialog, to allow use of a login and password plus use of a proxy server if desired.   The Web Login dialog is also handy for providing a Test button that can be used to test the connection.

Open as read-only

Open the data source read-only.  Usually has no effect with web servers since they are read-only in any event.

Cache data

Cache image tiles downloaded from the server while the project is open,  saved within the .map project itself in a Cache subfolder in the System Data hierarchy.

Save cached data between sessions

Save the cached data for the next time this project is opened,  within the .map project itself in a Cache subfolder in the System Data hierarchy.  Caution: checking this box can result in very large .map files when the image results of browsing web servers are all saved.  However, having such data cached in the .map is handy for offline browsing of the project.

API key

Provide a key that authorizes use of an API when connecting to a proprietary data source that requires such a key.

Create Data Source

Create the new data source in the project pane and close the dialog.

Edit Query

Launch the Command Window loaded with a query that creates the data source using the given settings.  A great way to learn how to use SQL to create data sources.

Cancel

 Exit the dialog without doing anything.

 

Web Login Dialog

When a Web Server: data source is the Type, the [...] button next to the Source box launches the Web Login dialog that allows specifying a server and connection characteristics.

 

dlg_web_login_create_nds.png

 

The above dialog provides the following controls:

 

Server

The connection string for the server.  This may be a simple URL or a very lengthy URL/connection string that embeds parameters such as keys that grant access or other parameters

Use login and password

Check this box for servers that require logging in with a login name and a password, providing the required name and password in the Login and Password boxes.

Use proxy server

Check this box when connecting through a proxy server.   The Proxy, Login, and Password boxes allow specifying the connection string to the proxy server as well as the login name and password required to use the proxy server.

API key

Provide a key that authorizes use of an API when connecting to a proprietary data source that requires such a key.

Application

A secondary application key or authentication code for those servers, such as here.com (also known as wego.here.com) that may require it.

Timeout

Specify a time in milliseconds to wait for connecting to the specified server.  Use 0 for the default timeout or specify whatever is the desired time to wait before giving up on the server.

Test

Press the Test button to try the connection using the specify parameters.   If successful, a Connection established information dialog will pop open.

Server Connection String

Server connection strings can contain much specialized information, such as required keys.  For example, a typical string to connect to a WMS server that provides topographic maps from New Zealands's LINZ / National Topographic office might be:

 

https://data.linz.govt.nz/services;key=ca62cf6a884a4f0cbfe5/wms/layer-767?service=WMS&version=1.1.1&request=GetCapabilities

 

We enter that into the Server box:

 

dlg_web_login_server_create_nds.png

 

(The above connection string has been edited so the key shown above is not accurate:  anyone can get their own key by registering on the LINZ website.)

Custom Web Servers

When choosing a Type of Webserver: imageserver, we can choose the Custom subtype to utilize a very wide range of web servers that provide image tiles using connection strings that contain some more-or-less typically encountered, but far from standard, control sequences.    Control sequences are tokens within curly { } brackets that are substituted during the action of the server with values that control the result.    To learn how a specific server uses control sequences, as well as learning other required information such as the tile size used and the zoom range supported, we must consult whatever documentation the server operator has provided.  

 

il_newdatasource_webserver_custom.png

Controls

Name

Name for the new data source, "Data Source" by default.  Specify a more  memorable name as desired.  If we forget the origin of a data source we can hover the mouse over the data source and a tool tip will provide connection information.

Type

Choose Web Server: imageserver and Custom in the secondary box below.

Source

A URL connection string that contains control sequences used by the server.  

[...] Browse button

Click to help populate the Source box.  Clicking the [...] Browse button will launch the Web Login dialog, to allow use of a login and password plus use of a proxy server if desired.   The Web Login dialog is also handy for providing a Test button that can be used to test the connection.

Open as read-only

Open the data source read-only.  Usually has no effect with web servers since they are read-only in any event.

Cache data

Cache image tiles downloaded from the server while the project is open,  saved within the .map project itself in a Cache subfolder in the System Data hierarchy.

Save cached data between sessions

Save the cached data for the next time this project is opened,  within the .map project itself in a Cache subfolder in the System Data hierarchy.  Caution: checking this box can result in very large .map files when the image results of browsing web servers are all saved.  However, having such data cached in the .map is handy for offline browsing of the project.

Tile size

Size of tiles issued by the server.  Always square, so 256 means tiles that are 256 pixels wide by 256 pixels high.  Some servers use 512 pixel tiles.   See the documentation for the server.

Zoom Range

0 is zoomed all the way out and 20 is zoomed all the way in to the closest zoom served by the server.  The closest zoom varies by server and must be entered correctly.   See the documentation for the server.

API key

Provide a key that authorizes use of an API when connecting to a proprietary data source that requires such a key.

Application key

A secondary application key or authentication code for those servers that require it.

Create Data Source

Create the new data source in the project pane and close the dialog.

Edit Query

Launch the Command Window loaded with a query that creates the data source using the given settings.  A great way to learn how to use SQL to create data sources.

Cancel

 Exit the dialog without doing anything.

Control Sequences

The Custom imageserver dataport understands the following control sequences, as typically used:

 

{x}, {y}

X and Y coordinates of a tile

{zoom}

Zoom level of a tile.  Zoom level is a numeric value similar to an image level but going in the reverse direction: an image level of 0 is the most detailed level, while a zoom level of 0 is the least detailed level supported by the server.   Some URL connection strings specified for servers use the control sequence token {z}.  Replace {z} with {zoom} in the URL to construct a URL to use as the  Source string.

{key}

API key, if entered by the user.

{keyApp}

Application key, if entered by the user.

{lon}, {lat}

Longitude and latitude of the center of the requested tile.

{switch:a,b,c}

Server selector for load balancing: Choices after the semicolon are available alternatives.

Examples

Dozens of examples using the Custom dataport have been published in the JOSM sources.map project that is available on the Release 9 Downloads page on the Manifold website.  Most of the web server data sources in that project are created from strings published in the JOSM sources page at https://josm.openstreetmap.de/wiki/Maps page, from which URL strings may be copied for use in the Source box of the Custom web server data source creation dialog, along with other parameters such as maximum zoom.  

 

Following are some examples of entries for various servers from the JOSM sources page.   These show selected tags and not all information for what sometimes are very long entries.   See the JOSM sources page for complete info.

 

OpenStreetMap using French cartographic style:

 

        <name>OpenStreetMap (French Style)</name>

        <type>tms</type>

        <url>http://{switch:a,b,c}.tile.openstreetmap.fr/osmfr/{zoom}/{x}/{y}.png</url>

        <max-zoom>20</max-zoom>

 

OpenStreetMap using German cartographic style:

 

        <name>OpenStreetMap (German Style)</name>

        <type>tms</type>

        <url>http://{switch:a,b,c,d}.tile.openstreetmap.de/tiles/osmde/{zoom}/{x}/{y}.png</url>

        <max-zoom>18</max-zoom>

 

In both the above cases we can copy the string between the <url> and </url> tags to use as our Source string.  In the case of the German style version we must change the Zoom range value in the New Data Source dialog from [0, 20] to [0, 18]

 

OpenTopoMap

 

        <name>OpenTopoMap</name>

        <type>tms</type>

        <url>http://tile.opentopomap.org/{zoom}/{x}/{y}.png</url>

        <min-zoom>3</min-zoom>

        <max-zoom>17</max-zoom>

 

We can copy the string between the <url> and </url> tags to use as our Source string.  We must change the Zoom range value in the New Data Source dialog from [0, 20] to [3, 17]

 

OpenCycleMap

 

        <name>OpenCycleMap</name>

        <type>tms</type>

        <url>https://{switch:a,b,c}.tile.thunderforest.com/cycle/{zoom}/{x}/{y}.png?apikey=7bd5ed2197cf4da29fa26de0ba6530cc</url>

        <max-zoom>22</max-zoom>

 

An example using an embedded API key, apparently issued by the originator for use by people creating features for OpenStreetMap.  We can copy the string between the <url> and </url> tags to use as our Source string.  We must change the Zoom range value in the New Data Source dialog from [0, 20] to [0, 22]

 

Some of the web servers in the JOSM sources.map project are licensed only for creating features in OpenStreetMap while others are completely unrestricted.  See the comments in the project for links that give complete info.

Cache Data Option

When creating new data sources for a web server data source we can check or uncheck the Cache data box (checked by default)  in the New Data Source dialog.   This option merits attention when creating data sources from which we might draw hundreds of megabytes of imagery or other data.

 

eg_imagesrv_tour01_48.png

 

Creating an imageserver data source with the Cache data option checked will cache all tiles received from the server while that project is open.   If in addition the Save cached data between sessions box is checked the tiles will be saved into the project file within the Project's System Data hierarchy in a sub-folder called Cache.

 

Web connections are often fast in many parts of the world but even so fetching tiles from a web server is not as fast as fetching them from local storage on local disc.   By caching tiles that we have seen as we pan and zoom within the images served by the web server we can provide faster performance, very much faster in the case of slower Internet connections, if we view a scene more than once.   

 

For example, if we view a scene and then zoom in to part of it, zoom back out and then zoom into another part of it, when we zoom back to views we have already seen if the tiles for that view are cached the system does not need to fetch them from the server over the web but instead it can fetch them much more quickly from cached storage.   If we save the project and then open it later, the cached tiles will still be in the project so any views, such as a zoom to fit followed by zooms into areas of interest, are likely to be viewable using tiles already in cache.

 

Another benefit of caching tiles is that the system does not have to re-fetch tiles when the same scene is viewed in the future.  When using tile servers operated by small organizations that can be a polite way to minimize the load we place on the organization's resources.

 

Improved rendering performance by using cache is wonderful, of course, but it comes at the cost of storing tiles within cache tables that are saved within the project if the Save cached data between sessions box is checked.   As we zoom into views and pan to different views the number of tiles we have seen can grow to where the data stored in cache can be hundreds of megabytes.

 

If we have plenty of free space on disk, terabytes, we might not care about that at all.  But if space is limited or if we want to send a project to someone else without also sending all of the cached tiles we should delete the cache.

 

A compromise is to check the Cache data box but to leave the Save cached data between sessions box unchecked.   In that case tiles will be cached for the duration of our session so if we zoom back and forth and pan between the same views we will get snappy performance without having to fetch and re-fetch the same tiles, but at the same time when we save the project and move on to another project the .map project file will not contain the possibly many megabyte of cached tiles.

Using Cache for Offline Viewing

If we check the Save cached data between sessions box,  all tiles that are served by the web server we are using will be saved within the .map project in the Cache folder within the System Data hierarchy.   When we save the .map project, all of those tiles will be saved within the .map file that is created.    If we spend a lot of time browsing, the cache can easily grow to hundreds of megabytes, since every time we pan and zoom within the display to a different view we might download another megabyte or so of tiles.  

 

The advantage of a saved cache is that if we re-visit the same view the system does not need to re-fetch those tiles.   If a tile for a given view is already in cache, even if we do not have an Internet connection, Manifold can still show it to us.  Instead of fetching it from the server through Internet, it simply pulls the tile from the saved cache.   We can exploit that effect to "load up" a project with many views from a web server, visiting locations and panning and zooming around in areas of interest, to fill up the cache with tiles from the various views we might want to see later on.  When we save the project it will be loaded with many tiles in cache.

 

If we later want to open and use that project in a location where we have no Internet, we can use the project as before, viewing tiles from cache.   For example, suppose we will be visiting an archaeological site in a location where there is no Internet and we would like to use Google Satellite imagery to guide us along dirt roads along our proposed track.   In our home or office where we have fast Internet, we can create a project that has a Google Satellite imageserver layer, along with layers showing approximate roads, the location of our archaeological site and so on.  

 

We spend some time browsing that project, panning and zooming the map in areas of interest.  For example, if we plan on using a particular access route we might zoom into that route and then pan the view along the route, moving the view from an access road through poorly marked terrain to our site.  Around our site we can zoom to various zoom levels and move the display around, capturing different views in cache.   We then save the project.

 

Out in the field where there is no Internet we can open the project and so long as we pan and zoom the map within the regions that we previously visited, with views and zoom levels like those we used to "load up" the project with cached tiles, the system will show us those tiles again.  It will seem as if we have live Internet, only faster, since everything will appear loaded from cache without any lag time at all.

 

Deleting Cached Tiles

To delete cached data for web servers that is created when the Save cached data between sessions box is checked we drill down into the System Data hierarchy and then delete the tables storing the tiles.

 

eg_imagesrv_tour01_44.png

 

Click onto the + icon next to the System Data folder to open it.

 

eg_imagesrv_tour01_45.png

 

Within the System Data folder we click open the Cache folder.   We see there are two tables there, one table for the street image tiles served by the Bing streets imageserver and one table for the satellite image tile served by the Bing satellite imageserver.

 

eg_imagesrv_tour01_46.png

 

With the Cache folder highlighted we click the Delete button or simply press the Delete key on the keyboard.

 

eg_imagesrv_tour01_47.png

 

The Cache folder and the two tables it contains are also deleted.   We can delete tables which cache tiles at any time without worries of de-synchronizing what is displayed.  The system adjusts automatically.  

 

Creating New Data Sources within Existing Data Sources

New data sources can be created either within the current .map project or within another data sources that is nested within the map project if that nested data source is writable.   If we have moved focus in the Project pane into another data source nested within our project then the Data Source box will be automatically populated with the name of that data source if it is writable.

 

For example, suppose we have a project that includes two data sources, one called France and the other called Germany, and both of those data sources were created from Manifold .map format files.   We can create anything within those external data sources that we could create in the current map project.  

 

il_nested_newdatasource01_01.png

We can right-click on the France data source so that has the focus and choose New Data Source.

 

il_nested_newdatasource01_02.png

The New Data Source dialog will open with an advisory at the top that we are creating the new data source within another data source called France, which is a manifold data sources.      We specify a useful name, choose a Type of file: manifold, browse over to the .map we want to use and then click the Create Data Source button.

 

il_nested_newdatasource01_03.png

 

That creates a new data source within the France data source.  If we expand it, we see that it contains all of the many items within the .map project file we used as a data source.  It even contains yet another nested data source, in this case a Virtual Earth hybrid image from a Virtual Earth web server.

 

Example: Create a New Data Source in a Folder

To see how the Path box works in the New Data Source dialog we can create a new data source within a folder in our project.

 

eg_create_nds02_01.png

 

Suppose we have a project we call Europe and within the project we have created a folder called Schengen Zone Countries.    We Right-click that folder and then in the context menus choose Create - New Data Source

 

eg_create_nds02_02.png

 

We enter a descriptive name, France, into the Name box and then use the [...] browse button to choose a .map file we have previously created that we wish to use as the data source.   We press Create Data Source.

 

eg_create_nds02_03.png

 

A new data source is created within the Schengen Zone Countries folder.     If we drill down into the newly created France data source and open one of the drawings we can see that we can reach into the France project through the data source.   Performance is identical, with zero loss, to having the drawing in our local project and not in a different project accessed as a data source.

 

 

eg_create_nds02_03a.png

 

 

We will save the Europe project.  Whenever we open the Europe project it will have within it the data source that we created which in turn links to the France project.   We will use this Europe project in the Example: Create a Data Source within an Existing Data Source topic.

 

Portals to Other Worlds

When we add a data source to a project we create, in effect, a portal to a different world.   When we open the data source to show the hierarchy within, the Project pane is giving us a view into the different world of that data source, where the rules that apply are the rules of that world.    For example, if we create a read-only data source, then we can open the hierarchy within that data source and view what is in it, but we cannot add components within that world and we cannot change components within that world.

 

il_portal_other_worlds01_01.png

 

Consider the above illustration, a section of a screenshot shown in the Example: Create a Drawing from Read-Only Geocoded CSV File topic.   The portion of the hierarchy shown within the read-only data source is read-only.  We cannot create any new components in that portion of the Project pane because that portion resides in the different world of a read-only data source that links in a read-only .csv file.  We can open the Dots table and read data from it and the table can participate in queries by providing data, but we cannot change anything in that table.  For example, we cannot change the schema of the table by adding fields or indexes to it and we cannot edit values within records in that table.  

 

The region bordered in green and marked Read / Write is part of the world of the local project.   We can do whatever we like in that region, such as right-clicking there and using Create - New Drawing to create a new drawing in the project.

 

il_portal_other_worlds01_02.png

 

Consider the above illustration, a section of a screenshot shown in the Example: Create a Data Source within an Existing Data Source topic.  It shows worlds within worlds.  The hierarchy within the Europe data source is a world within the .map file called Europe.map that was linked in as a data source into our Project.   The hierarchy our Project pane shows is a window into the world of that other .map file.    It shows that within the Europe map file's world there are yet three more worlds, a linked in data source called England and two more linked in data sources called France and Germany, all of them linked in from other .map files.  

 

One of the worlds, that is, data sources, has been expanded so we can see a window into that world as well, the world of the Germany.map file.    In the above illustration all of the linked in worlds are themselves Manifold Release 9 .map files that have been linked in without any read-only restrictions, so we can work with them with full Manifold capabilities.    The rules which apply within those worlds are the same as the rules which apply in the local project, since Manifold is seamless in that way.  

 

However, if the world of, say, the Germany data source was that of a database such as SQL Server or PostgreSQL, then whatever rules which such databases apply within their own projects, for example, what data types are available,  would apply within that hierarchy.

 

Notes

Passwords - Some data sources to which we might want to connect may use passwords.   Normally such data sources are databases such as Oracle or SQL Server and the dialogs to connect to them using the Database: option as the Type will allow specifying a password.    In some cases file databases such as Microsoft Access MDB might also require a password.   To specify a password we should connect using the Database: oledb option, which utilizes a dialog that allows specifying a password, and not  using the File Database: mdb option which is a simpler option that does not allow specifying a password.

 

tech_ravi_sm.png

Tech Tip:  Some web servers to which we might connect using WMS, WMTS, WFS or other connection standards can be incredibly unresponsive or, despite allowing a connection will not allow data to be fetched.   If pressing the Test button results in a successful connection, or if layers available are displayed when we expand the data source, but then when we try to display an advertised layer nothing happens, we can open the Log Window to see what is going on.

 

Here are a few lines of log window information for a happy session, connecting to a WMS server in New Zealand run by the ever-efficient LINZ:

 

Web request: (root)::[Data Source] (GetCapabilities) (0.578 sec, 13.0 KB)

Web request: (root)::[Data Source]::[layer-767] (GetMap) (1.255 sec, 47.9 KB)

Render: [Data Source]::[layer-767 Image] (1.561 sec @1.0)

 

In contrast, a few lines of log window information for an unhappy session, attempting to connect to the WMTS server run by the French cartographic service IGN but getting no image for a cadastral parcels layer:

 

Web request: (root)::[IGN]::[CADASTRALPARCELS.PARCELS_bdparcellaire_b] (GetTile) (0.070 sec, 0 B)

*** (root)::[IGN]::[CADASTRALPARCELS.PARCELS_bdparcellaire_b] (GetTile) The remote server returned an error: (403) Forbidden.

Render: [IGN]::[CADASTRALPARCELS.PARCELS_bdparcellaire_b Image] (0.315 sec @1.0)

 

From the above we see an empty cadastral parcels image was rendered because the remote server refused to provide tiles in response to the GetTile request, returning a 403 error Forbidden.    Finding out why a web server refuses service can involve some detective work.  

 

The server might require a special key for access, a key we are trying to use may be out of date, a login and password may be required for some resources, the connection string may have been copied and pasted inaccurately or, of course, in the case of IGN the server may be refusing to work to show solidarity with striking  railroad workers or air traffic controllers.

 

Keep in mind that just because a web server offers tiles via various standards like WMTS or whatever does not mean that it will be fast or even reasonably tolerable.  Some web servers are painfully, incredibly slow and will take many seconds or even minutes per tile.   When images do not appear or fill in with painful slowness a look at the Log Window can show the problem is not our Internet connection or Manifold, but just incredibly slow response from the web server.

 

Three Letter Extensions - Most file names in Windows end in what is called a three letter extension, which is usually three letters at the end of the file name following a dot . character.    The three letter extension is one way Windows at times (but not always) keeps track of what a file is supposed to be.    Unfortunately, by default Windows hides the three letter extensions of files and instead tries to associate files with icons of whatever program is normally associated with that file.  This is confusing when working with the many file formats that Manifold and similar products utilize.  

 

Therefore, please turn off the hiding of extensions by Windows.  A typical way to do so in most versions of Windows would be from Windows Explorer, choose Tools - Folder options, press the View tab and then in the Advanced Settings pane ensure that the Hide extensions for known file types is unchecked.  Press the Apply to Folders and then press OK.  You will then be able to see extensions such as .map and others.  See also the Essay on three letter extensions and why the default hiding of them by Windows is such a bad thing.

 

See Also

That YX Thing - An essay discussing how it is that so much spatial data claiming to use EPSG standards fails to honor the standard, and thus why Manifold must provide a check box to deal with such errors.

 

Web Servers

 

New Favorite Data Source - Choose from a drop down list of Favorite data sources, or choose Edit Favorites to launch the Favorites dialog for data sources.

 

Favorite Data Sources - View and edit the Favorites dialog for data sources, showing various data sources that may be added to the project with a single click from the list provided to the New Favorite Data Source menu.

 

Example: Spectacular Images and Data from Web Servers - A must see topic providing a gallery of views illustrating how Manifold can use web servers such as imageservers and other free resources to provide a seemingly endless selection of spectacular background maps, satellite images and GIS data with nearly zero effort.

 

Example: An Imageserver Tutorial - An extensive tutorial showing step by step how to add new data sources that are imageservers, how to show them as layers in a map, how to create a new drawing that matches the projection of the map and how to trace over what is seen in an imageserver layer to create an area object in the drawing.

 

Example: Create a New Data Source from a Manifold Image Server - Manifold image server modules are snippets of code which use the Manifold Image Server Interface (ISI) to automatically fetch image tiles from popular image servers like Virtual Earth, Wikimapia, Yahoo!, Google Maps, Yandex and many others. Image servers can provide street maps, overhead satellite imagery, combinations of streets and satellite imagery and other data as well.  Using Manifold Image Servers is one of the most popular Manifold features.

 

Example: Create a New Data Source from a MAP File - Create a new data source from an existing Manifold .map project file.   This is the classic way to nest projects, one calling another, to create libraries of data and projects.   Access to nested projects has effectively zero performance loss and the links within projects take up effectively zero space so we can create huge constellations of data at our fingertips.

 

Example: Create a Data Source within an Existing Data Source - When a data source is writable, for example, if the data source is a Manifold .map file, we can work within that data source as if it were at the top level of our project.   For example, we can create a new data source that is nested within the existing data source.   This example shows how.

 

Example: Create and Use New Data Source using an MDB Database - This example Illustrates the step-by-step creation of a new data source using an .mdb file database, followed by use of SQL.  Although now deprecated in favor of the more current Access Database Engine formats, .mdb files are ubiquitous in the Microsoft world, one of the more popular file formats in which file databases are encountered.  

 

Example: Modify GPKG Geometry with SQL then Add Drawing - This topic provides a "Hello, World" example that shows a simple, but typical, task involving spatial data.  We will take a country-sized data set in GeoPackage (GPKG) format and change all areas in the data to the boundary lines for those areas and then save those boundary lines as a new table.  We add a spatial index to the table and create a new drawing to visualize the new table.

 

Example: Trace an Area in a Map over an Image Background - In a map with a drawing layer above an image layer (served dynamically by an imageserver), create an area object in the drawing by tracing over the outlines of something seen in the image layer below

 

Example: Style Applied to an Image Server Image - Because the Edit - Style dialog simply changes the way an image is displayed and not the data, it can operate on read-only data served by various web servers such as WMS REST servers.    In this example we look at every detail of creating a data source using a WMS REST  server and then manipulating the appearance of the display with Style.  We will connect to a WMS server that provides LiDAR data in various forms, including as terrain elevation.