GIS work often involves reading and writing information stored in database servers or data files. Manifold allows us to work with information in databases in many different ways and allows us to connect to all database and data file technologies in common use in Windows systems.
Examples of connecting to data include:
· Connect to a SQL Server database server within Database Console to examine the contents of that database.
· Upload an image from a Manifold project into an Oracle Spatial DBMS.
· Link a surface into a Manifold project from a MySQL DBMS.
· Link a drawing from a SQL Server database.
· Import a table from an Access .MDB file into a Manifold project.
· Link a table from an Access .MDB file into a Manifold project.
· Import a table from a .CSV text file.
Manifold provides two pathways to connect to data:
· For simple connections to simple data, we can use the File - Export, File - Import and File - Link commands to connect to simple file types such as CSV, DB, DBF, HTML, MDB, WK and XLS directly by choosing the desired file type in the Files of Type box in the dialogs. This is similar to how we would work with such files in typical Windows applications.
· For more complex data sources or more sophisticated uses, when launching the File - Export, File - Import and File - Link commands we would choose Data Sources () in the Files of Type box in the dialogs. This launches a standard Manifold dialog, the Data Source dialog, as a simple user interface for connecting to databases and data files. The Data Source dialog can also connect to simple file types as above and is used to connect to those simple file types for more sophisticated uses, such as linking a drawing from a table stored in a file. The Data Source dialog is always used when linking a drawing or linking a surface.
The Data Source dialog remembers a list of previously-used data sources and allows point and click connection without requiring users to remember what can be complex connection settings or obscure locations on hard disk.
The remembered list of previously-used data sources will be available within the Data Source dialog no matter what part of Manifold needs to connect to a data source. For example, it is frequently the case that if we import data from a database we might also want to export data to that same database. It is therefore very convenient to have that database connection already in the Data Source dialog's list whether we launch commands from the File - Import or the File - Export menus.
The first time we use a data source we will have to add it to the list in the Data Source dialog. Thereafter, the data source will be remembered in the list.
The Data Sources dialog works with all data source connection technologies supported by Manifold. The Data Sources dialog works with the Database Console, the Administrator Console and is also available when importing, linking and exporting components such as drawings, images and surfaces via the File - Export, File - Import, File - Link and similar dialogs. Choosing Data Sources () in the Files of Type box in such dialogs will invoke the Data Sources dialog.
Data Sources
There are generally two types of data sources we use with Manifold:
· File-based data sources - These are simply data files, such as .CSV (comma separated values) text files, .XLS (Excel), .DBF (dBase II / FoxBase), .MDB (Access) or other files. Some such data files can be very simple, such as plain, human-readable ASCII text files while other such data files, such as .MDB, can be more complex and require software that understands their internal organization to utilize. What file-based data sources have in common is that they all store data in files within Windows and applications that want to utilize the data in those files will open, read and write those files using some suitable connection methodology. Manifold can connect to all popular data file formats using either a connection technology specific to that data file such as, for example, Microsoft Jet for standard Microsoft file types such as Access .MDB or Excel .XLS, or a generic connection technology such as ODBC, OLE DB or ADO .NET.
· Database servers - DBMS servers are software products like Oracle, SQL Server, DB2, PostgreSQL, MySQL and similar that provide information through software connections. When we connect to a DBMS server we may not know exactly how the server stores information since we never actually connect to a specific file. Instead, the DBMS server provides whatever information we need. Manifold can connect to all popular DBMS servers using either a connection technology specific to that server such as, for example, OCI (Oracle Call Interface) for Oracle, or a generic connection technology such as ODBC, OLE DB or ADO .NET.
Either of the above types of data source can be added as a data source within the Data Source dialog. In addition, a third way of specifying a data source connection, DSN and UDL files, can also be specified as a data source.
DSN and UDL files do not themselves store data but instead are a way of saving within a file the information required to establish an ODBC or an OLE DB connection with a given data source, which could be file-based storage or a database server. A DSN connection describes a connection using ODBC and a UDL file describes a connection using OLE DB. Opening the DSN or UDL file within Manifold establishes the connection specified by that file.
Using the Data Sources Dialog
On first use the Data Sources dialog has no data sources listed. We first add a data source to the list. We can then double-click the data source to connect. The Data Sources dialog will remember data sources for future use, including in future Manifold sessions.
To Add a Data Source
1. Choose an operation, such as pressing the ... button in the Database Console, that launches the Data Sources dialog.
2.
Click on the Add Data Source button. This launches the Connect To dialog.
3. In the Connect To dialog choose the connection method desired in the Files of type box and proceed through the dialogs for that connection method to add the data source.
To Connect to a Data Source
1. Launch the Data Sources dialog.
2. Double-click on a data source, or, click on a data source in the list to highlight it and press OK.
To Delete a Data Source
1. Launch the Data Sources dialog.
2. Click on a data source in the list to highlight it.
3.
Click on the Delete button.
Data Sources Dialog Commands
|
|
Add Data Source - Add a new data source. |
|
|
Delete - Delete highlighted data source. |
|
(contents pane) |
Displays a list of data sources, showing the name and type of connection for each. |
|
Name |
Name to use for the highlighted data source. |
|
(connection string) |
Connection string to use for the highlighted data source.
|
|
|
Click to edit connection string using the standard dialogs for that connection technology. |
Any PASSWORD and PWD parameters used in a connection string will be masked in the connection string to protect security.
Data Source Connections for Spatial DBMS
In addition to the usual roster of ADO .NET, OLE DB, and ODBC connections Manifold Enterprise Edition and higher editions provide specialized native DBMS connection choices that are used when adding a new data source using a spatial DBMS in the Data Source dialog.

|
DB2 Data Sources |
Connect to IBM DB2 equipped with the DB2 Spatial Extender using IBM's native spatial connection technology. |
|
Connect to Oracle using OCI, the Oracle Call Interface to utilize Locator or Oracle Spatial facilities. |
|
|
PostgreSQL Data Sources |
Connect to PostgreSQL equipped with the PostGIS spatial extender using native PostgreSQL connection technology. Connections made using this data source will have any password required masked so that later usage will not inadvertently expose the password. |
|
SQL Server Data Sources |
Connect to Microsoft SQL Server 2008 spatial DBMS facilities using native SQL Server 2008 spatial connection technology. |
Users sometimes ask if there is a special connection when using SQL Server 2005 as a spatial DBMS with the Manifold Spatial Extender for SQL Server installed. There is no need for a special connection because Manifold when connecting to SQL Server using any OLE DB connection will recognize and automatically use spatial indices managed by the spatial extender when the Manifold spatial extender has been installed.
Example: Create a Data Source for SQL Server Express
Let's add a connection to a local SQL Server Express data source to the Data Sources dialog. This is a straightforward connection for using SQL Server 2005 and not a native connection to SQL Server 2008 spatial.

We begin by launching Database Console and then clicking the [...] button to specify a data source.

Click the Add Data Source button in the Data Source dialog.

In the Connect To dialog choose OLE DB Data Sources () in the Files of type box.

In the Data Link Properties dialog's Provider tab we choose the SQL Native Client and press Next.

In the Connection tab we specify the settings used for our local SQL Server Express installation. In the case of this example we use the default data source name of (local)\SQLEXPRESS and Windows NT integrated security, which is the default recommended by Manifold's SQL Server Express installation instructions.
Pressing the Test Connection button is almost always a good idea at this point, to make sure there is nothing grossly wrong (such as a typographical error) in the information provided. When we are happy the connection is live, we press OK.

Back in the Data Sources dialog we see that a new data source has been added using the default name for the connection. It is usually wise to enter some more memorable name that will help us recognize data sources in a more useful manner. This is easy to do.

In the Name box we can enter the friendly name we would like to use to identify this data source. Given the sophistication of data connection technologies available in Windows we might have many different data sources we use. For example, we might have a connection to a DBMS on a different machine that we wish to name County parcel records on main server or some other useful name that will help us immediately recognize that data source.
In this manner we can add more data sources. When done adding data sources, we can highlight the one desired and press OK.

Back in the Database Console we can see we have connected to the local SQL Server Express database. This database has very little data in it, only three drawings. Note that Database Console will display the friendly name we assigned to this data source.
Example: Create a Data Source for an Access .MDB File
Let's add a connection to an Access .mdb database file. We'll connect to the sample Northwind Traders file distributed with many editions of Access.
We begin by launching Database Console and then clicking the [...] button to specify a data source.
Click the Add Data Source button in the Data Source dialog.

In the Connect To dialog choose MDB Files in the Files of type box. Navigate to the folder and file desired and double-click on the file, in this case the nwind.mdb file.

This adds another OLE DB data source, since Manifold automatically connects to files such as .mdb using Microsoft's built-in OLE DB provider for such files. To avoid confusion we should change the name of this data source to something more recognizable.

We change the Name to Northwind Traders Sample MDB and press OK.

The result is that Database Console connects to the Northwind Traders database in the nwind.mdb file.
Example: Create a Data Source for Oracle Express
Let's add a connection to a local Oracle Express data source to the Data Sources dialog.
We begin by launching Database Console and then clicking the [...] button to specify a data source.
Click the Add Data Source button in the Data Source dialog.

In the Connect To dialog choose Oracle Data Sources () in the Files of type box. This will connect using OCI, the Oracle Call Interface native to Oracle.

In the Oracle Data Source dialog we provide the Server name, the User Name and the Password. Settings shown are the default settings recommended in Manifold's installation instructions for Oracle Express. As always, pressing the Test button to make sure we have entered the password and other information correctly is a good idea. When we are happy the connection is live, we press OK.

Back in the Data Sources dialog we see that a new data source has been added using the default name for the connection and showing that it is a native Oracle connection. We will add a friendly name.

In the Name box we enter Local Oracle Express. This will help us remember that this particular Oracle data source is a connection to the local DBMS, which will be useful if in the future we create connections to Oracle databases on different machines. Press OK.

Back in the Database Console we can see we have connected to the local Oracle Express database. (We could use some work on the filter settings in Database Console for this particular connection to suppress display of all the various DBMS objects that are not Manifold components.)
Linking Drawings or Surfaces
The Data Source dialog is always used when linking drawings or surfaces. For example, if we would like to link a drawing from an MDB file (such as when creating a linked drawing from a geocoded table), we must first add the MDB file as a data source in the Data Source dialog and then link the drawing from that Data Source.

In addition, when linking a drawing or a surface the Data Source dialog will launch with a "built-in" data source of This Project. Choosing that data source will connect to the project and make available components within the project from which the drawing or surface can be linked. The primary usage will be linking a drawing or surface from a table or query within the project.
Connection Technologies
There are many different connection technologies in use in the Microsoft world that we can use to connect to data sources.
When connecting to a native spatial DBMS we must always use the native connection technology if one is available. For example, when connecting to Oracle databases we should always use OCI, the Oracle Call Interface.
Use OLE DB to connect to databases when using generic Manifold spatial indices for spatial DBMS work or when connecting to SQL Server 2005 using the Manifold Spatial Extender for SQL Server or when connecting in a general way to databases. The disadvantage of using OLE DB is that the dialogs required to initiate a connection are more complex. Given that the Data Source dialog remembers previous data sources added to the dialog's list, that is not inconvenient because once a connection is set up it can be used with a simple double-click in the Data Sources dialog.
Connecting via ADO.NET
Manifold can also import or link data from ADO.NET data sources. Manifold includes a generic importer that is listed as ADO.NET Data Sources() in the Files of Type box when creating a new data source in the Data Source dialog. In addition, Manifold includes four dedicated importers designed for importing or linking data from the standard ADO.NET providers installed by the .NET Framework: ODBC, OLE DB, Oracle and SQL Server.
The generic import prompts for a .NET DLL (in the Assembly box) with an ADO.NET provider, the provider class, and the connection string.
The ADO.NET ODBC importer displays the standard ODBC data source dialog. The ADO.NET OLE DB importer displays the standard OLE DB data source dialog. The ADO.NET Oracle importer prompts for a server name and login credentials (integrated security or a specific login and password). The ADO.NET SQL Server import prompts for a server name, login credentials (integrated security or a specific login and password), and optional database name.
Some UDL (Universal Data Link), DSN, OLE DB and ODBC data sources may be dependent on the presence of third party database systems or ODBC drivers in the Windows system. For example, to connect to Btrieve databases, we will need a Btrieve ODBC driver on our system. To connect to Oracle databases, we will need an Oracle database server to which we can connect. Manifold automatically installs a variety of OLE DB providers that will connect to most popular data sources.
Important: Tables linked from ADO .NET data sources are always read-only. If read-only access is acceptable, ADO .NET is often the fastest possible connection and should be used in preference to ODBC or OLE DB for that reason. If read-only access is not acceptable, then OLE DB should be used if possible.
Oracle Call Interface
When connecting to Oracle data sources Manifold can use the Oracle Call Interface (OCI), the native Oracle interface for exchanging data. Using the native interface provides better performance and allows using features not exposed through generic database interfaces such as ODBC, OLE DB and ADO .NET.
Exchanging data with Oracle data sources via OCI automatically maps Manifold geometry columns into Oracle SDO_GEOMETRY data. SDO_GEOMETRY values representing collections of geometric entities are not supported.
Data Link Properties Dialog
When configuring a data source, Manifold uses the standard Microsoft OLE DB data source Data Link Properties dialog to allow connection to OLE DB data sources. It's a lot easier to use than it looks.

Choose the OLE DB data link type desired. Note that in Microsoft's world there are often many different ways to access the same type of data. For example, Manifold can open any Access .mdb file directly, or we can open an .mdb file by using the Microsoft Jet 4.0 OLE DB Provider. If we've installed any "Office" software or other database capable software, we will also have an ODBC driver on our system that can open Access .mdb files as well. So, we could use the OLE DB Provider for ODBC Drivers to open an ODBC data source for .mdb files.
After choosing the provider, click the Next button to move to the Connection tab.

The Connection tab will be pre-loaded with the right connection parameter choices for the type of OLE DB provider selected. In the illustration above, we've chosen the provider for ODBC Drivers and so we see the right connection options for connecting via ODBC. The data source name list box will be pre-loaded with the ODBC data sources currently configured on our system.

If we had chosen the OLE DB Provider for SQL Server and then pressed Next, the Connection tab would be loaded with the right choices to connect to a SQL Server database via OLE DB.
Provide the necessary information for the provider chosen and then press OK. The Advanced tab shows various advanced options for the specified provider, and the All tab provides a summary of the data link properties specified.
Sophisticated database systems such as SQL Server or Oracle may be configured in sophisticated ways. The information in the Connection tab and, possibly, the Advanced tab reflect the sophistication of such systems. If you are unfamiliar with the specific expectations of the SQL Server or Oracle or other provider that you wish to use, you may have to consult with the database administrator in charge of the installation to determine the correct settings to use.
Connecting Via ODBC
ODBC as a database connection technology preceded OLE DB, so there are more ODBC drivers for different types of databases than there are OLE DB drivers. Manifold System connects to ODBC data sources through whatever ODBC drivers are installed on the computer.
ODBC is easy to use if we remember that the slightly tedious part is creating a new "source." Some ODBC sources are quite simple and consist of specifying what type of database driver to use (dBase, Access, etc.). Other ODBC sources, such as connecting to SQL Server via ODBC, are more complex and require what type of driver to use, which database is to be opened and from which machine or server it is fetched. In the case of some systems, such as SQL Server and Oracle, there are both OLE DB and ODBC drivers.
Set up ODBC data sources using the Windows ODBC Data Sources administrator dialog that is called by clicking on the ODBC Data Sources icon in the Windows Control Panel. This icon is located in the Administrative Tools folder in the Control Panel in Windows 2000 and subsequent Windows editions.
Using UDL Files and OLE DB Data Sources
A Microsoft UDL file is simply a pre-packaged shortcut to an OLE DB data source. We create a UDL file as noted below. We can then use OLE DB data link properties dialogs to configure the data link that will be used by the UDL file. Note that when opening an OLE DB data source or creating a UDL file we end up using exactly the same OLE DB dialogs.
Why bother with creating a UDL file if doing so requires us to use the OLE DB dialogs anyway? It's not really necessary with the Data Sources dialog, which can remember an OLE DB connection perfectly well. However, creating a UDL file can serve as a shortcut for other applications to connect to the same data source.
Creating and Using a UDL File
Creating a UDL file is a standard Microsoft procedure that is documented in Windows (search for "UDL" in Windows Help). Suppose we have an Excel spreadsheet that contains a table and we would like to connect to that table using a UDL file. Our spreadsheet is called MySheet.xls. We would proceed as follows:
1. In Windows Explorer, right click and choose New - Text Document. Create a text document called MySheet.udl. Windows will complain about changing the extension from .txt to .udl. That's OK.
2. Double-click on MySheet.udl to open the Data Link Properties dialog.
3. Configure the Data Link Properties dialog as shown in the illustration below. Press OK. The UDL will now contain a data link to the Excel spread sheet.
4. Launch Manifold and choose File - Import - Table and then open MySheet.udl. Choose the table desired.

UDL files are normally used to automate more complicated connections than shown above. They may be used to provide user credentials such as a user name and password or to use a complex connection string when required. Windows 2000, Server 2003 and XP all include documentation for creating UDL files as part of Windows help. Other versions of Windows may not include this documentation, although the capability is there if you have installed any one of many packages that install the Microsoft data access routines that enable universal data links.
Performance
The internal Manifold database engine used to maintain imported tables is usually substantially faster than external database systems, and is matched or exceeded only by the very fastest enterprise class DBMS products such as Oracle or SQL Server.
Manifold's facility with Microsoft data types as well as installation remarks for some Windows versions admonishing users to install Jet service packs occasionally leads to the misconception that Manifold internally uses Jet for database management. That is not the case.
Although Manifold includes a copy of Microsoft's "Jet" database engine (the same used in Access), Jet is not used to maintain tables that are imported into Manifold. Jet is used only as an accessory to handle external files. All imported tables are managed using the internal Manifold database engine and all queries are executed using the Manifold SQL engine, both of which are purpose-built systems engineered for the specific demands of fast geometry / attribute work in GIS.
When linking external tables into Manifold one ends up using whatever database engine or system is associated with those external tables. Performance will therefore almost always decrease when using linked tables from simple, consumer-style data sources (such as Access or Excel tables) as compared to the speed of imported tables.
Linked tables will be as fast as or faster than internal Manifold tables when the external tables are very large and the provider is a sophisticated engine such as SQL Server or Oracle using a fast connection such as ADO .NET (or, in some circumstances, OLE DB) for SQL Server or OCI for Oracle. Even if they are not as fast as Manifold's dedicated engine for smaller databases, enterprise database systems such as SQL Server nonetheless are still very fast. Therefore, it is not likely that any performance difference between imported tables or linked tables will be noticed with smaller tables when enterprise class engines such as Oracle or SQL Server are used.
When using external DBMS tables, use the fastest connection possible, which may vary depending on the DBMS in use. For example, when connecting to Microsoft's SQL Server, ADO.NET is faster than OLE DB and OLE DB is faster than ODBC. ADO.NET is much faster than ODBC. However, ADO .NET is generally read-only, which limits the utility of that technology. Consider that connecting to a remote SQL Server can take much longer than connecting to a local SQL Server. For example, suppose that connecting to a table inside the Manifold project takes 1 unit of time. A very rough guide to equivalent times to access the same table using different connection and DBMS technologies is listed in the accompanying table. Clearly, it is very unwise to connect to remote SQL Server installations using ODBC.
|
Time |
Source for Table |
|
1 |
Native table (inside the project) |
|
8 |
Linked from a local MDB file |
|
10 |
Linked from a local SQL Server via ADO.NET |
|
40 |
Linked from a local SQL Server via OLE DB |
|
10 |
Linked from a remote SQL Server via ADO.NET |
|
250 |
Linked from a remote SQL Server via OLE DB |
|
6000 |
Linked from a remote SQL Server via ODBC |
The default connection to Oracle servers is always OCI, so Oracle connections always run at the fastest possible connection speed unless we manually force the connection to use an alternative, such as OLE DB or ODBC.
See the Performance Tips topic for specific tips to maximize table and query performance.
Working with SQL Server and Oracle
Very important: When working with server-based OLE DB providers such as SQL Server and Oracle, users are strongly encouraged to maintain primary keys in all tables linked into the Manifold project. A side effect of how such servers interact through OLE DB is that if the table does not have a primary key, performance will be greatly reduced.
Linking to Oracle Tables
The default connection to Oracle tables is OCI, which greatly simplifies life with Oracle. Connect using the Oracle Data Sources type and all will be well.
However, if for some reason we need to connect using OLE DB or ODBC to an Oracle Server, we should make sure to follow these tips:
§ Always set the Allow saving password option. If this option is not set, the system will connect for the first time but not at all other times. The intricacies of the OLE DB / ODBC interaction that takes place within the Microsoft OLE DB Provider for ODBC Drivers are such that it is almost never possible for the Manifold to jump into the middle of the connection process and prompt the user for a username and password.
§ Do not use the native Oracle ODBC driver. Use the Microsoft OLE DB Provider for ODBC and use Microsoft ODBC for Oracle. The native Oracle drivers are limited to forward-only cursors (which is by far the simplest and the lowest performance type of database cursor allowed in ODBC drivers) and thus are unusable in Manifold or many other programs, such as Microsoft Access. Future editions of Manifold will add special routines to allow use of the Oracle driver.
Working with OLAP
OLE DB drivers allow connections to many data sources besides simple database tables. For work with OLAP, for example, Microsoft's SQL Server Decision Support Services (the Microsoft OLAP package) has an OLE DB that can be used to import or link a "table" that pulls data from the OLAP cube. The driver supports Microsoft's MDX language that can be used to fetch information from OLAP cubes. The MDX language is very rich and thus allows sophisticated work with OLAP cubes. Needless to say, the driver works fine with Manifold.
When Logins and Passwords are Required
When linking tables via a connection to some database providers such as Oracle or SQL Server the connection may fail without proper user credentials if security settings in the database provider require a username and password. In such cases Manifold will raise a login dialog that allows specification of a user name and password.
The dialog includes a Remember username and password checkbox that is off by default. Check this box to save the provided user name and password with the table within the .map file. This will enable automatic linking using these credentials the next time the .map file is opened. Note: the user name and password will be saved with the table in encrypted form within the .map file. It is strongly recommended that any such .map files containing your user credentials be protected by setting appropriate Windows access privileges on the .map file or the folder that contains it.
Tech Tips
Manifold can use all of the standard Microsoft DBMS connection technologies, such as ADO .NET, to connect to data sources. There are a huge number of possibilities with such connection technologies that are documented by Microsoft and other educational resources for standard Microsoft facilities. As this documentation is aimed at Manifold and is not intended as a general educational resource for standard Microsoft facilities it will not attempt to duplicate the many educational resources already available for those standard Microsoft technologies.
If you would like to utilize databases such as SQL Server or Oracle and you would like to connect to them using technologies such as OLE DB or ADO.NET, there are very many books on those subjects that provide full details, tips and other useful information on using these standard connection technologies.
The data source selection dialog automatically recognizes data sources which do not support storing drawings, images or surfaces. For example, we cannot store images into a .txt file. Attempting to select a data source which does not support storing components of a certain type during the export of a component of that type displays a warning message below the list of data sources and disables the OK button.
Notes
In a perfect world, every OLE DB provider or ODBC driver or external DBMS they serve that is installed on our systems would work without error. Regrettably, in real life the various database drivers and providers installed in our systems will contain bugs. Even in the case of the high quality drivers provided by Microsoft, one only need read the Microsoft Knowledge Base and the release notes for various Microsoft service packs to see that many bugs have been identified in such drivers. Manifold uses Microsoft drivers and OLE DB providers to connect to various types of database tables. If there are errors in the Microsoft drivers, such errors will affect Manifold as they would any other application.
If you suspect a database driver bug, take time to research the issue within the online Microsoft Knowledge Base. A useful experiment might be to import the table instead of linking to it and seeing if the problem persists. If it goes away, that is evidence that the problem lies in the external database system or is a bug in the drivers for that system. To reduce the likelihood of bugs in external software make sure you have applied the latest service pack for the external database system you are using.
When running SQL within Manifold queries, one is using the Manifold SQL engine. When executing SQL within the Database Console one is using whatever SQL is the native SQL of the external database system. One should be aware that SQL implementations in various database systems can contain numerous bugs. For example, even as well crafted an SQL as Jet SQL used within Microsoft's Access products contains numerous bugs. If an SQL bug occurs within the Database Console, the bug should be tracked down with the vendor of the external database system being used.
See Also