Importing and Linking

Most often the data we will want to use in a Manifold project originates in some source outside of Manifold  We often have a choice whether to bring that data into the Manifold project by importing the data into the project, or working with the data left in the original source by linking that data into the project.  Manifold can read virtually every format and data source in existence.   See the Big List of Formats and Data Sources topic for the huge list.   

Importing Data

Importing data from a file copies data from the originating file format and stores it within the project as a local component, leaving the original file unchanged and preserving no connection to the originating file.   Until the project is saved the data is maintained in a temporary Manifold format file.  When the project is saved to a .map project file the data will be stored in that .map file.   



Importing large files can take a long time because the imported data will be analyzed and stored in special, pre-computed data structures within the Manifold file that allow subsequent reads and writes to be very fast.  It pays to be patient with such imports as once the data is imported and stored within a Manifold project file access to that data will usually be far faster than it was in the original format.    Once imported the data will open instantly thereafter..


Importing data will create tables and other components within the project.   Depending on the data that is imported, importing will create components within the project to display the data in appropriate form.   For example, vector data imported from typical GIS file formats will create one or more tables for the data as well as one or more drawings that display that data in visual form.   Importing data brings it into the project within Manifold infrastructure under full Manifold management for maximum performance.     See comments in the How to Edit a Single File topic.


Keep in mind that when we import into a project we break all connections between whatever it was we imported and the former format.   The Manifold dataport which handles the import will automatically do all necessary conversions and will built all necessary Manifold infrastructure so whatever we import can thereafter be a genuine Manifold thing.   When we import an image it is no longer a JPEG, TIFF or PNG image and instead becomes a Manifold image.     


Beginners who say "Oh, I'm working on this JPEG image in Manifold and need to change the style..." are making a classic novice error: an image in Manifold is not a JPEG image.  It is a Manifold image and a Manifold image only.   Whatever we do with it has nothing at all to do with JPEG.  Some day we may want to export it as a JPEG, and if we decide to do that the Manifold dataport doing the export will take the Manifold image and convert it into whatever form is required to save as a JPEG.   



Manifold jargon - Dataports are Manifold modules that interact with data sources to provide meaning and structure and access to the data in those data sources in different ways.   A simple dataport knows how to import or to link to data in a specific format.   A more complex dataport knows how to interpret ESRI-arranged data within an Oracle database as an Arc Personal GDB data source.  

Linking Data 

Linking data from a file establishes at least a read connection and if possible a read/write connection to the originating file format.  



Linked components such as tables, drawings or images will appear within the project pane hierarchy and, for fully read/write connections, can be used as seamlessly as if they were stored within the project, but all linked data remains stored in the original file and is not copied or otherwise moved into storage within Manifold.   If the data connection to the linked file is read/write we can edit the data, within whatever limits are imposed by the file format. If the data connection is read-only, we will not be able to edit the data.    tech_ravi_sm.png


Caution: Editing data in a linked file (other than .map project files) that is writable usually will immediately change the data in that file.


Saving a project will also save all of the links for linked components in the project.   Linked components take no storage space in a saved .map project file because all that is saved in the project file is a link to the file holding the data.   We should keep that mind when moving the .map project file.   If we send a copy of the .map file to a friend we are only sending whatever data is in the file in local components.   Unless our friend also has a copy of the files that are linked into the project when he or she opens the project file in Manifold the system will not be able to find the files that are supposed to be linked into the project because they will not exist on our friend's machine.


Linking a file into a project can be a very quick process for small files or a significantly slower process for larger files.    But once a file has been linked Manifold's use of technology like MAPCACHE can make it very fast to view and to edit.   For all that if we bring data into a project by linking a non-Manifold file format into our project we will be limited by all of the limitations of that file's format.    See comments in the How to Edit a Single File topic.

Limitations on Linked Data

Linking a file into a project is different from importing because the data remains saved in the original format outside of Manifold.   When linking a file into a project the Manifold dataport doing the linking will do its best to provide as close a simulation as possible of the comparable Manifold component, for example, providing virtual tables for JPEG images that are linked so that it seems we are working with a Manifold image and not a JPEG.   But depending on the format being linked that only goes so far.   For example, if we link an ESRI shapefile into Manifold it will look like a Manifold drawing but because shapefile format does not allow a mix of object times we will not be able to mix areas, lines and points within that drawing the way we could with a genuine Manifold drawing.

Cache and Linked Data

Cache in Manifold helps us get around some of the limitations of linked data.  When linking an image or other data into Manifold the Link dialog provides an Options button at the bottom of the dialog that allows us to choose cache options.  Using cache provides better performance and allows us to launch Assign Initial Coordinate System in the Contents pane to specify the correct initial projection for linked data.




Above the Open and Cancel buttons is a report of the current settings of Cache, with settings of partial, persistent in the illustration above.



Caching data within the project will save a copy of data that is brought in from the external file within the project.   It is like a virtual import that, so long as the data within the external source has not changed, uses the data that has already once been brought into the project instead of re-fetching it again from the external source.    Checking the Save cached data between sessions box will save cached data within the .map project file so the next time that .map project file is opened what was cached from the prior session will still be available for use.   If the external source has changed since the last session, then cached data will be discarded as obsolete and fresh data will be fetched to refill the cache.


Cache provides two primary ways of improving performance:



Using or not using cache is not a big deal for small data, like small images, but when images or other data are gigabytes in size cache can provide dramatic improvements in speed.  That is why the Cache data box is checked by default.  


Cache also provides some of the benefits of imported images.   For example, if the Cache data box is enabled we will be able to launch Assign Initial Coordinate System in the Contents pane to specify the correct initial projection for linked data.   If we have turned off cache we will not be able to assign the initial projection.

Create a New Data Source

il_colosseum_sm.pngCreating a new data source establishes a dynamic connection to a data source, which could be a database server such as Oracle or PostgreSQL, a file database such as slqlite or a Microsoft .mdb file, a web server providing data using some protocol such as WMS or a Manifold imageserver module or even a plain, ordinary file format.    


Do not miss the use of image servers, as discussed in topics like the Example: Spectacular Images and Data from Web Servers example topic.


Creating a new data source using a file format is equivalent to using the File - Link command to link that file into a project (the capability is duplicated within the New Data Source dialog for user convenience) along with the option to connect in read-only mode.  When connecting to file formats new data sources are normally used with data sources based on .map projects, with file databases or with server-based data sources such as a DBMS or a web server source.



Caution: By default data sources are opened in read / write mode if they are writable. When we edit data stored in a .map file the .map file is not changed until we do a Save, but in contrast, when we edit data in a data source such as a database system, any such changes made usually will immediately change the data in that data source without waiting for a Save.  That is expected behavior to database people but it may come as a surprise to someone used to a document editing model who may expect "nothing will change in the originating source until I do a save."


To protect data sources from inadvertent changes we can open them for read-only access by checking the Open as read-only box in the New Data Source dialog when creating the data source.  Opening a data source read-only in many cases will also allow other processes or users to simultaneously also open those data sources read-only to view them at the same time we view them, if the data source allows such multi-user, read-only access.



When we create a new data source Manifold adds the data source to the project pane using a gray cylinder icon to indicate it has not yet been opened. When we open the data source, say, by double-clicking on it,  Manifold begins scanning it using background threads and changes the icon to a yellow cylinder. Some data sources, like large Enterprise class database servers, can contain hundreds or thousands of tables and it can take some time to scan the data source to collect their names for display in the project pane.




During the scanning process a small triangle, like the "play" symbol on audio players, appears in the lower right corner of the data source yellow cylinder icon.  




When scanning is done the triangle scanning sub-icon goes away.  Scans are often so quick we might never notice the scanning sub-icon.  A yellow cylinder icon in the project pane for the new data source indicates the data from that source is stored outside the project within the data source.




If we create a data source that is read-only the icon will include a small black lock to indicate we cannot change data within that data source.



Note: In a Manifold project the System Data folder contains the two special mfd_meta and mfd_root tables that Radian uses to operate the project.  When we link data into a project or when we create a new data source from some non-Radian file, database system or other data source, Radian will show a System Data folder within that link or data source that contains mfd_meta and mfd_root tables as if they existed within the link or data source.  However, those tables do not actually exist within the non-Radian link or data source.   


In the case of such linked sources they are synthesized as virtual tables by Radian for Radian's use in operating that link or data source.  They are visible only to Radian and can be used only within Radian.  They do not actually exist in the data source and the data source's native query engine, if one exists, will not be able to see them.  For example, if we link a shapefile into a Radian project we won't create two tables called mfd_meta and mfd_root within any of the files that make up a "shapefile."


Radian will use the System Data folder for a linked data source to save other housekeeping infrastructure.   For example, Radian dataports for WFS, WMS, WMTS, OSM and other web servers which cache data will create cache tables within a Cache folder within the data source's System Data folder.


Advantages and Disadvantages of Imports

As convenient as it is working with data "in place" in whatever format or source it was originally, formats and sources vary widely in their technical characteristics such as speed, data capacity, robustness in the face of network or Windows glitches or ability to handle parallel processing that lets you use the full power of your computer.  


Importing data into Manifold avoids such problems, but it comes with some advantages and disadvantages.

Two Disadvantages

Importing data into Manifold has two disadvantages:



Many Advantages

In exchange for the above two disadvantages, importing data into Manifold has many advantages:












In an era when disk space is basically free but our time is more valuable than ever, it makes sense to trade off slightly larger files to get the above benefits.   It also make sense to tolerate slow imports of very large files on a one-time basis.  Most files import quickly enough.    Experienced Manifold users tend to move as much of their data as possible into Manifold project files, using .map files to archive data that originally may have been in other formats.


An Import Strategy


Tech Tip:  When you first install Manifold System, begin a systematic plan of converting your larger files into Manifold format during off hours.  Set a reminder on your computer to import a file just before lunch hour and just before your work day ends.  When you return, save the project in Manifold .map format.  Do that on a regular basis and before you know it, all of your big files will have been imported into native Manifold storage, ready to open instantly.


Apply the same plan to link into a Manifold project any files like ESRI shapefiles or MapInfo files you may some day want to edit "in place."  Start linking them into your project when you will be away from your desktop and then save the .map file when you return.  That will build a .MAPCACHE file, a potentially long process for a big shapefile.


If in advance you systematically import or link big files and then save the results as Manifold .map files, when you need to work with that data you'll be able to open the project instantly and work with lightning speed.   Enjoy life more.  Do your imports in advance.


See Also

Getting Started


Projects and .map Files


User Interface Basics


Assign Initial Coordinate System


Where Manifold Data is Stored