Importing and Linking

Most often the data we will want to use in a Manifold project originates in some source outside of Manifold  We often have a choice whether to bring that data into the Manifold project by importing the data into the project, or working with the data left in the original source by linking that data into the project.  Manifold can read virtually every format and data source in existence.   See the Big List of Formats and Data Sources topic for the huge list.   

 

Importing a file brings data from the file into the Manifold project.   No ties to the originating file remain and the data within the project, such as a drawing or image or table, is a purely Manifold component with all Manifold facilities available.

 

To import a file:

 

  1. In the main menu choose File - Import.

  2. In the Import dialog, navigate to the folder containing the desired file.

  3. Double-click the file to be imported.

 

In most cases Manifold will automatically determine the correct file type to use for the importing dataport based on the "three letter" file extension.  Sometimes when the file type cannot be determined we may have to choose the specific file type from the long list available, click on the file to choose it, and then click Open.

 

To import data from a linked data source:

 

  1. Open the data source in the Project pane, usually by clicking the + box next to the cylinder icon that indicates a data source.

  2. Navigate within the data source hierarchy to the component desired.

  3. Ctrl-click desired components to highlight them.

  4. Press the Copy icon in the Project pane toolbar, or press Ctrl-C.

  5. Click anywhere in the main part of the Project pane, that is, outside of the data source hierarchy.

  6. Press the Paste icon in the Project pane toolbar, or press Ctrl-V.

 

The above makes a local copy of the data within the project.  It is the same as importing the desired components, as opposed to linking them and leaving them in storage within the external data source.   When copying components such as drawings and images, do not forget to copy both the drawing and the table from which it takes data.  

 

Linking a file leaves data in storage within the file.   It only appears to be within the Manifold project.  Although Manifold does a good job of giving the appearance that such linked data can be used as if it were a true Manifold component, because external file formats generally are far more limited than Manifold native storage not all Manifold facilities might be available with linked data.

 

To link a file:

 

  1. In the main menu choose File - Link.

  2. In the Link dialog, navigate to the folder containing the desired file.

  3. Double-click the file to be linked.

 

In most cases Manifold will automatically determine the correct file type to use for the linking dataport based on the "three letter" file extension.  Sometimes when the file type cannot be determined we may have to choose the specific file type from the long list available, click on the file to choose it, and then click Open.

 

Creating a new data source is a more sophisticated way of linking data.   It uses a more comprehensive dialog than a simple File - Link and so is the way we link data from databases, where optional settings such as login or password for the database might be required.

 

To link data as a new data source:

 

  1. In the main menu choose File - Create - New Data Source.

  2. In the New Data Source dialog provide a Name for the new data source if the default is not desired.

  3. Choose the Type of data source from the long list available.

  4. Choose the sub-type of data source when those are available.

  5. The Source box will be automatically loaded for some data sources, for example, connection URLs to Bing or Google servers.   For other sources, use the [...] browse button to navigate to the file or data source connection as required.

  6. Set option boxes that appear as desired, for example, checking Open as read-only if the data source should be kept read-only.

  7. Press Create Data Source.

 

See the File - Create - New Data Source topic for details.

"Unlinking"

Suppose we have linked a file and now we would like to "unlink" it, that is, to import it?  That is easy to do: Copy the desired components in the data source and then Paste them into the main part of the drawing.

 

Unlinking a linked file or data source:

 

  1. Open the data source hierarchy, and drill down through any folders to the desired drawing and table, or other desired components.

  2. Right-click onto the desired drawing and table, or other desired components.

  3. Press Copy.

  4. Click into the main part of the project, that is, outside of the linked data source.

  5. Press Paste.

  6. If the linked data source is no longer needed, delete it.

 

Importing Data

Importing data from a file copies data from the originating file format and stores it within the project as a local component, leaving the original file unchanged and preserving no connection to the originating file.   Until the project is saved the data is maintained in a temporary Manifold format file.  When the project is saved to a .map project file the data will be stored in that .map file.   

i_file_import.png

 

Importing large files can take a long time because the imported data will be analyzed and stored in special, pre-computed data structures within the Manifold file that allow subsequent reads and writes to be very fast.  It pays to be patient with such imports as once the data is imported and stored within a Manifold project file access to that data will usually be far faster than it was in the original format.    Once imported the data will open instantly thereafter..

 

Importing data will create tables and other components within the project.   Depending on the data that is imported, importing will create components within the project to display the data in appropriate form.   For example, vector data imported from typical GIS file formats will create one or more tables for the data as well as one or more drawings that display that data in visual form.   Importing data brings it into the project within Manifold infrastructure under full Manifold management for maximum performance.     See comments in the How to Edit a Single File topic.

 

Keep in mind that when we import into a project we break all connections between whatever it was we imported and the former format.   The Manifold dataport which handles the import will automatically do all necessary conversions and will built all necessary Manifold infrastructure so whatever we import can thereafter be a genuine Manifold thing.   When we import an image it is no longer a JPEG, TIFF or PNG image and instead becomes a Manifold image.     

 

Beginners who say "Oh, I'm working on this JPEG image in Manifold and need to change the style..." are making a classic novice error: an image in Manifold is not a JPEG image.  It is a Manifold image and a Manifold image only.   Whatever we do with it has nothing at all to do with JPEG.  Some day we may want to export it as a JPEG, and if we decide to do that the Manifold dataport doing the export will take the Manifold image and convert it into whatever form is required to save as a JPEG.   

 

tech_tina_sm.png

Manifold jargon - Dataports are Manifold modules that interact with data sources to provide meaning and structure and access to the data in those data sources in different ways.   A simple dataport knows how to import or to link to data in a specific format.   A more complex dataport knows how to interpret ESRI-arranged data within an Oracle database as an Arc Personal GDB data source.  

Linking Data 

Linking data from a file establishes at least a read connection and if possible a read/write connection to the originating file format.  

i_file_link.png

 

Linked components such as tables, drawings or images will appear within the project pane hierarchy and, for fully read/write connections, can be used as seamlessly as if they were stored within the project, but all linked data remains stored in the original file and is not copied or otherwise moved into storage within Manifold.   If the data connection to the linked file is read/write we can edit the data, within whatever limits are imposed by the file format. If the data connection is read-only, we will not be able to edit the data.    tech_ravi_sm.png

 

Caution: Editing data in a linked file (other than .map project files) that is writable usually will immediately change the data in that file.

 

Saving a project will also save all of the links for linked components in the project.   Linked components take no storage space in a saved .map project file because all that is saved in the project file is a link to the file holding the data.   We should keep that mind when moving the .map project file.   If we send a copy of the .map file to a friend we are only sending whatever data is in the file in local components.   Unless our friend also has a copy of the files that are linked into the project when he or she opens the project file in Manifold the system will not be able to find the files that are supposed to be linked into the project because they will not exist on our friend's machine.

 

Linking a file into a project can be a very quick process for small files or a significantly slower process for larger files.    But once a file has been linked Manifold's use of technology like MAPCACHE can make it very fast to view and to edit.   For all that if we bring data into a project by linking a non-Manifold file format into our project we will be limited by all of the limitations of that file's format.    See comments in the How to Edit a Single File topic.

Limitations on Linked Data

Linking a file into a project is different from importing because the data remains saved in the original format outside of Manifold.   When linking a file into a project the Manifold dataport doing the linking will do its best to provide as close a simulation as possible of the comparable Manifold component, for example, providing virtual tables for JPEG images that are linked so that it seems we are working with a Manifold image and not a JPEG.   But depending on the format being linked that only goes so far.   For example, if we link an ESRI shapefile into Manifold it will look like a Manifold drawing but because shapefile format does not allow a mix of object times we will not be able to mix areas, lines and points within that drawing the way we could with a genuine Manifold drawing.

Cache and Linked Data

Cache in Manifold helps us get around some of the limitations of linked data.  

 

dlg_link_options01.png

 

When linking an image or other data into Manifold the Link dialog provides a Save cache option at the bottom of the dialog that allows us to cache data from a linked file.   Using cache provides better performance and allows us to launch Assign Initial Coordinate System in the Contents pane to specify the correct initial projection for linked data.

 

dlg_link_options.png

 

Using File - Create - New Data Source allows more sophisticated control over cache option.  

 

Caching data within the project will save a copy of data that is brought in from the external file within the project.   It is like a virtual import that, so long as the data within the external source has not changed, uses the data that has already once been brought into the project instead of re-fetching it again from the external source.    Checking the Save cached data between sessions box will save cached data within the .map project file so the next time that .map project file is opened what was cached from the prior session will still be available for use.   If the external source has changed since the last session, then cached data will be discarded as obsolete and fresh data will be fetched to refill the cache.

 

ico_nb_arrow_blue.png

  Caution: When creating data sources based on web servers, like a Google image server, turning on the Save cached data between sessions option will save in cache a copy of every image tile fetched form the server.  That can result in astonishingly fast performance when next we browse that same region in the project, because all tiles will be local and will not need to be fetched over Internet from the server, but it will also mean that the size of the project can become very large (hundreds of megabytes or many gigabytes) very quickly.   Most Manifold users will leave that check box OFF unless they want to use that data when off-line.

 

Cache provides two primary ways of improving performance:

 

 

Using or not using cache is not a big deal for small data, like small images, but when images or other data are gigabytes in size cache can provide dramatic improvements in speed.  That is why the Cache data box is checked by default when creating data sources on regular file formats.  It is not checked by default when creating data sources on web servers.

 

Cache also provides some of the benefits of imported images.   For example, if the Cache data box is enabled we will be able to launch Assign Initial Coordinate System in the Contents pane to specify the correct initial projection for linked data.   If we have turned off cache we will not be able to assign the initial projection.

Create a New Data Source

il_colosseum_sm.pngCreating a new data source establishes a dynamic connection to a data source, which could be a database server such as Oracle or PostgreSQL, a file database such as slqlite or a Microsoft .mdb file, a web server providing data using some protocol such as WMS or a Manifold imageserver module or even a plain, ordinary file format.    

 

Do not miss the use of image servers, as discussed in topics like the Example: Spectacular Images and Data from Web Servers example topic.

 

Creating a new data source using a file format is equivalent to using the File - Link command to link that file into a project (the capability is duplicated within the New Data Source dialog for user convenience) along with the option to connect in read-only mode.  When connecting to file formats new data sources are normally used with data sources based on .map projects, with file databases or with server-based data sources such as a DBMS or a web server source.

tech_angus_sm.png

 

ico_nb_arrow_blue.png  Caution: By default data sources are opened in read / write mode if they are writable. When we edit data stored in a .map file the .map file is not changed until we do a Save, but in contrast, when we edit data in a data source such as a database system, any such changes made usually will immediately change the data in that data source without waiting for a Save.  That is expected behavior to database people but it may come as a surprise to someone used to a document editing model who may expect "nothing will change in the originating source until I do a save."

 

To protect data sources from inadvertent changes we can open them for read-only access by checking the Open as read-only box in the New Data Source dialog when creating the data source.  Opening a data source read-only in many cases will also allow other processes or users to simultaneously also open those data sources read-only to view them at the same time we view them, if the data source allows such multi-user, read-only access.

il_wikimapia_munich_sm.png

 

When we create a new data source Manifold adds the data source to the project pane using a gray cylinder icon to indicate it has not yet been opened. When we open the data source, say, by double-clicking on it,  Manifold begins scanning it using background threads and changes the icon to a yellow cylinder. Some data sources, like large Enterprise class database servers, can contain hundreds or thousands of tables and it can take some time to scan the data source to collect their names for display in the project pane.

 

icon_data_source_scanning.png

 

During the scanning process a small triangle, like the "play" symbol on audio players, appears in the lower right corner of the data source yellow cylinder icon.  

 

icon_data_source.png

 

When scanning is done the triangle scanning sub-icon goes away.  Scans are often so quick we might never notice the scanning sub-icon.  A yellow cylinder icon in the project pane for the new data source indicates the data from that source is stored outside the project within the data source.

 

icon_data_source_locked.png

 

If we create a data source that is read-only the icon will include a small black lock to indicate we cannot change data within that data source.

il_esri_physical_sm.png

 

Note: In a Manifold project the System Data folder contains the two special mfd_meta and mfd_root tables that Radian uses to operate the project.  When we link data into a project or when we create a new data source from some non-Radian file, database system or other data source, Radian will show a System Data folder within that link or data source that contains mfd_meta and mfd_root tables as if they existed within the link or data source.  However, those tables do not actually exist within the non-Radian link or data source.   

 

In the case of such linked sources they are synthesized as virtual tables by Radian for Radian's use in operating that link or data source.  They are visible only to Radian and can be used only within Radian.  They do not actually exist in the data source and the data source's native query engine, if one exists, will not be able to see them.  For example, if we link a shapefile into a Radian project we won't create two tables called mfd_meta and mfd_root within any of the files that make up a "shapefile."

 

Radian will use the System Data folder for a linked data source to save other housekeeping infrastructure.   For example, Radian dataports for WFS, WMS, WMTS, OSM and other web servers which cache data will create cache tables within a Cache folder within the data source's System Data folder.

Advantages and Disadvantages of Imports

As convenient as it is working with data "in place" in whatever format or source it was originally, formats and sources vary widely in their technical characteristics such as speed, data capacity, robustness in the face of network or Windows glitches or ability to handle parallel processing that lets you use the full power of your computer.  

 

Importing data into Manifold avoids such problems, but it comes with some advantages and disadvantages.

Two Disadvantages

Importing data into Manifold has two disadvantages:

 

 

Many Advantages

In exchange for the above two disadvantages, importing data into Manifold has many advantages:

 

 

 

 

 

 

 

 

 

 

 

In an era when disk space is basically free but our time is more valuable than ever, it makes sense to trade off slightly larger files to get the above benefits.   It also make sense to tolerate slow imports of very large files on a one-time basis.  Most files import quickly enough.    Experienced Manifold users tend to move as much of their data as possible into Manifold project files, using .map files to archive data that originally may have been in other formats.

An Import Strategy

tech_tina_sm.png

Tech Tip:  When you first install Manifold System, begin a systematic plan of converting your larger files into Manifold format during off hours.  Set a reminder on your computer to import a file just before lunch hour and just before your work day ends.  When you return, save the project in Manifold .map format.  Do that on a regular basis and before you know it, all of your big files will have been imported into native Manifold storage, ready to open instantly.

 

Apply the same plan to link into a Manifold project any files like ESRI shapefiles or MapInfo files you may some day want to edit "in place."  Start linking them into your project when you will be away from your desktop and then save the .map file when you return.  That will build a .MAPCACHE file, a potentially long process for a big shapefile.

 

If in advance you systematically import or link big files and then save the results as Manifold .map files, when you need to work with that data you'll be able to open the project instantly and work with lightning speed.   Enjoy life more.  Do your imports in advance.

Projections / Coordinate Systems

The terms projection and coordinate system are used as synonyms by Manifold.   

 

Getting it right on import:  When importing data, Manifold will use whatever coordinate system the source says should be used for that data.   Most modern formats used in GIS will automatically specify the coordinate system to be used.   But older formats won't do that, so we have to assign the coordinate system when we import from those formats.

 

ico_nb_arrow_blue.png The Component panel of the Contents pane tells us at a glance if a coordinate system has been assigned OK.   When a data source specifies the coordinate system to use, Manifold displays the coordinate system in black color.  If the coordinate system is shown in red color, we must click the coordinate system picker button and choose Assign Initial Coordinate System.

 

il_contents_component01_01.png  il_contents_component01_02.png

 

Manifold shows the coordinate system in black color, as in the Latitude / Longitude coordinate system seen above at left, when the component has been created with coordinate system information or if it has been imported from a format that specifies the coordinate system to use.   When a component has been created without specifying the coordinate system or it has been imported from a format that does not specify projection information, Manifold will show the default Pseudo Mercator coordinate system as a placeholder in red color.   

 

ico_nb_arrow_blue.pngRed color in a coordinate system name warns us that we must assign an initial coordinate system:  Click the coordinate system picker icon and use the Assign Initial Coordinate System dialog to specify what projection should be used.  Do that immediately after import whenever a coordinate system appears in red text.  tech_ravi_sm.png

 

Tech tip:   What projection is used by data can be difficult to determine sometimes.    When downloading data from the web or other sources, always look for any accompanying notes or metadata information that says what projection is used for the data.  If we have no idea what projection is used and the file is older data, a good guess would be Latitude / Longitude.   For example, shapefiles that do not have a .prj file attached to specify what projection they use are often in Latitude / Longitude projection.

 

For a complete, step-by-step example, see the Example: Assign Initial Coordinate System topic.

Portals to Other Worlds

When we add a data source to a project we create, in effect, a portal to a different world.   When we open the data source to show the hierarchy within, the Project pane is giving us a view into the different world of that data source, where the rules that apply are the rules of that world.    For example, if we create a read-only data source, then we can open the hierarchy within that data source and view what is in it, but we cannot add components within that world and we cannot change components within that world.

 

il_portal_other_worlds01_01.png

 

Consider the above illustration, a section of a screenshot shown in the Example: Create a Drawing from Read-Only Geocoded CSV File topic.   The portion of the hierarchy shown within the read-only data source is read-only.  We cannot create any new components in that portion of the Project pane because that portion resides in the different world of a read-only data source that links in a read-only .csv file.  We can open the Dots table and read data from it and the table can participate in queries by providing data, but we cannot change anything in that table.  For example, we cannot change the schema of the table by adding fields or indexes to it and we cannot edit values within records in that table.  

 

The region bordered in green and marked Read / Write is part of the world of the local project.   We can do whatever we like in that region, such as right-clicking there and using Create - New Drawing to create a new drawing in the project.

 

il_portal_other_worlds01_02.png

 

Consider the above illustration, a section of a screenshot shown in the Example: Create a Data Source within an Existing Data Source topic.  It shows worlds within worlds.  The hierarchy within the Europe data source is a world within the .map file called Europe.map that was linked in as a data source into our Project.   The hierarchy our Project pane shows is a window into the world of that other .map file.    It shows that within the Europe map file's world there are yet three more worlds, a linked in data source called England and two more linked in data sources called France and Germany, all of them linked in from other .map files.  

 

One of the worlds, that is, data sources, has been expanded so we can see a window into that world as well, the world of the Germany.map file.    In the above illustration all of the linked in worlds are themselves Manifold Release 9 .map files that have been linked in without any read-only restrictions, so we can work with them with full Manifold capabilities.    The rules which apply within those worlds are the same as the rules which apply in the local project, since Manifold is seamless in that way.  

 

However, if the world of, say, the Germany data source was that of a database such as SQL Server or PostgreSQL, then whatever rules which such databases apply within their own projects, for example, what data types are available,  would apply within that hierarchy.

 

See Also

Getting Started

 

Projects and .map Files

 

Exporting

 

User Interface Basics

 

File - Import

 

File - Link

 

File - Create - New Data Source

 

Assign Initial Coordinate System

 

Where Manifold Data is Stored