Projects and .map Files

i_smallmfdicon.png

.map project files, called projects, are the native "documents" for Manifold, just like .doc files are the native documents of Microsoft Word and .psd files are the native documents of Adobe Photoshop.    In Manifold, the File - Open command will open an existing Manifold .map project file and the File - Save command will save a Manifold project in .map project file format.

 

Manifold stores everything it does in a project file in Manifold .map file format.    A .map project file is a very fast, very rich file database that can store large, complex entities like maps, drawings, scripts, queries, tables and much more, including links to databases in DBMS servers like Oracle or PostgreSQL, and connections to web data sources providing terabytes of data.   A .map project file can have thousands of entities in it and itself be hundreds of gigabytes in size.  

 

Manifold .map project format is an operational format: it is designed for everyday work prioritizing maximum speed and to enable cascading nesting of projects with zero performance loss.  For archival storage and exchanging data purposes, we can use File - Export Project to export any Manifold project into Manifold's archival storage format, .mxb format.   MXB is slower to write and slower to open than MAP and is cannot be used operationally for nesting projects, but it is a highly compressed and robust format that minimizes file size, perfect for archival storage and for minimizing the size of files for transmission.   MXB is also useful in ensuring maximum data reliability.  See the For Maximum Reliability topic.

Projects

A project is the collection of stuff we are working on in any given Manifold session.   It is all the different things we have in our Manifold workspace in that session.   The Project pane to the right of the Manifold desktop will show the contents of the project.  See the View - Panes - Project topic.

 

 

 

A new, blank project:

 

i_desktop_med2.png

 

Manifold always launches with a project, a new blank project called Project1 if not opened with an existing project file.    A new project is empty but for a System Data folder containing tables Manifold uses to manage the project.

 

i_system_data_closed.png

Clicking the + icon opens the System Data folder to show the two special tables it contains.   

 

i_system_data_open.png

We do not need to work with the System Data tables during routine Manifold use.   However, since they are small, simple tables and they always are at hand we can use them to illustrate examples in this documentation.

 

A previously created project:

 

i_desktop_med_populated_project.png

In the illustration above we have used File - Open to open a previously-saved project called Aus Hydro.map.    It all comes back, including a data source that is a Microsoft Bing image server that is ready to go. tech_angus_sm.png

 

Tech tip: Within the Project pane most Windows moves we know, like Copy and Paste, will work.  For example, we can Copy a table and Paste it within a folder.  We can drag and drop a table into a folder to move it there.  One modification to a standard Windows key: drag and drop in Windows is a move command and Ctrl-drag and drop is a copy command.   In Manifold, drag and drop also is a move command but Shift-drag and drop is a copy command.

Folders

We can organize what is in our project using folders, like the System Data folder seen above, a good idea if our projects include many components.   The Project pane itself includes useful features, like filtering, to help us zero in on the components we want in a project that may contain very many.

Components

When we add stuff to our project that stuff will appear in the Project pane as various items, called components. Components are items such as tables, images, scripts, data sources and so on depending on what sort of data is added.    The project pane will show all components in our project with an icon next to each to indicate the type of component.

 

il_project_pane_components.png

 

Tables are the center of everything in Manifold.   All data in Manifold is stored in tables in one form or another.  That's good because it means we can get at anything with SQL.   Nothing is hidden.  

 

Other components such as images or drawings are visualizations of data stored within some table.  Map components show other visual components, such as images, drawings and labels, as layers within a map window.

 

icon_table.png

Tables - Where all data within Manifold is stored in one form or another.  Opening a table window shows a table in classic row and column presentation.

icon_drawing.png

Drawings - A visual display of vector data stored within tables using a variety of geometry types. The same data can be seen in multiple different drawings using different styles, and the same drawing can be open in multiple windows with different selections and views in each.

icon_data_source_cylinder.png

Data Sources - Data linked into a project from an external source, which could be a file, a file database, a database serve or a web server, covering virtually every database and file type encountered in database or GIS work.

icon_image.png

Images - A visual display of raster data stored within tables as tiles using a variety of different data types and channel combinations.   All rasters are called images in Manifold even those which are terrain elevation or other non-photographic data.

icon_labels.png

Labels - Text annotations automatically created from the fields of a drawing's table.

icon_layout.png

Layouts - Layouts are compositions on virtual sheets of paper, allowing us to create documents showing maps or other components as PDF documents or hard copy printed sheets.

icon_map.png

Maps - Consist of a stack of layers where each layer is a drawing, image or labels component.

icon_query.png

Queries - Written in SQL, queries manipulate data and projects, for example, creating new tables and other components, performing calculations,  altering the structure of databases and tables, and extracting, editing and analyzing subsets of data.

icon_script.png

Scripts - Manifold includes built-in support for scripting in eleven different languages with six to eight always available and the remainder easy to install.  Scripts provide custom capabilities and can automate virtually anything.

icon_comments.png

Comments -  Text saved and displayed in a comments window provides a simple but  convenient way to save notes about a project or to save text as a scratchpad.

icon_folder.png

Folders - We can organize our project by using folders.

Linked Components and Local Components

Components we see in the Project pane may be a link to data stored somewhere else or they may be a local component for which the data is stored within the project file.   Linked components don't take up any space in the .map file that stores the project.   They're just pointers to the file or data source somewhere else where the data is stored.   Local components are stored in the project's .map file, which will grow in size as we add local components to a project.   .map files can get very big if we add large components to them.   

 

i_link_or_import.png

 

icon_data_source_cylinder.pngA yellow database cylinder marks a linked file or data source in the Project pane.  The yellow database cylinder tells us that everything within that hierarchy is coming into the project from some source outside of the project.  The source will also have its own System Data folder with information on the linked source.   In the above example the Products table seen under the products hierarchy is coming in from some data source outside of the project.   In contrast, the bronze image and the bronze Tiles table are both local components that are stored within the project file.

 

When we use File - Save to save a project we write that project into a file database stored in Manifold .map file format.  The .map file we create saves all the data for local components as well as saving links, Project pane folder hierarchies and other housekeeping information for the project.  The next time we open the project file in Manifold we will get all of that back, with the Project Pane showing components and folders organized exactly like they were when we saved the project.

 

Manifold project files can save large amounts of data, hundreds of gigabytes, and provide performance second to none in database applications.  That makes Manifold itself a fine choice for a file database system, using Manifold .map format project files for storage, without being forced to use an external DBMS just because the tables, images, drawings or other data we would like to store and to work with are very large.

 

At the same time Manifold is exceptionally adept at working with and very fast at connecting to DBMS servers like Oracle, PostgreSQL, DB2 or SQL Server, so fast that Manifold will take full advantage of the best performance those systems can provide.   That makes Manifold a great console for working with multiple different DBMS databases at once, with big data stored locally in the project file, or with big data stored in external DBMS servers or both at the same time.

 

Manifold can also link other, previously created Manifold project files into a project with no loss of performance compared to importing the data into local storage within the project.  For example, the products link seen in the illustration above is a Manifold .map project file called products that was linked into the project.   Manifold can then work with such linked-in projects with zero loss of performance no matter how big the linked-in project may be, just as if the contents of those linked-in projects were saved locally within the session's own project file.   Working with linked project files is so fast that we would never guess they were linked files from performance alone if we did not see in the project file they were marked with a yellow database cylinder.

Getting Data into a Project

There are many ways to get data into project:

 

Open an existing Manifold .map project file or Manifold .mxb archive file.    

Use File - Open to open a Manifold .map project file or a Manifold .mxb archive file.  

 

 

 

 

Large .map files in pre-Radian format can take a long time to be converted into Radian .map file format.  Be patient in such cases as once the project is saved by Manifold as a Radian .map file it will thereafter open instantly. 

 

Create new components.   

We can use the File - Create menu (or right-click into the Project pane and choose from the context menu) to create a new component such as a table or a drawing and then add data manually  to that component, or add data programmatically, taking data from some process that generates data.   The simplest way to add data manually is to create a table, specifying what fields we want in the table and to then proceed to add records one by one manually to the table.  We are not going to manually add a billion records to a table but a few hundred are easy to add and some organizations over time manually  will create tables with thousands of records.  

 

We can also create a drawing and then manually create objects in that drawing. Manifold System provides exceptionally rich editing capabilities for drawings.  The DBMS-centric subset of Manifold, Radian Studio, is not a GIS or CAD editor so editing facilities within Radian Studio for drawings are limited.   But for many purposes even the limited capabilities within Manifold are more than enough to allow viewing of drawings and the creation of new objects, for example, by tracing over what is seen in an imageserver.

Copy and Paste between Manifold sessions.  

If we launch more than one Manifold session we can open different projects with those sessions and then Copy and Paste between projects.   Because Manifold projects open instantly they make very good storage containers for collections of data.   Users will often have terabytes of folders on disk that are full of Manifold projects that contain their data holdings.   The same .map format is used for all Radian-based applications, so Manifold System can open and work with .map projects created and saved by Radian Studio.

 

We can use Manifold projects by linking them into our project or by popping the project open in a separate Manifold session and then copying and pasting from it to our current session. The best way to get data from one Manifold project into another is to create a data source for that other project within the session where we are working.   That is generally faster than making copies using the Windows Clipboard.   

tech_ravi_sm.png

 

Tech tip: When copying records from a table and pasting into another table, to add records to the destination table we should use the Ctrl-Shift-V command for a  Paste Append.   A paste append will automatically avoid collisions between existing and pasted values of the mfd_id key field used in tables that contain data for drawings and images because the pasted records will automatically be given new, unique mfd_id values.  If we neglect to use Ctrl-Shift-V we might not be able to paste any records because mfd_id values must be unique: if the records we would like to paste have any mfd_id values that already exit in the destination table they will not be pasted into that table.

Write scripts and programs to create new data or to acquire it.

We can utilize a variety of automated means to create new components and to populate them with data.   We can use SQL to create new tables and other components, either from existing ones or as a result of computations.   We can use SQL or scripting or a combination to connect to external sources, for example, external sensors such as GPS devices or cameras which generate data, and to capture that data within our project.

Saving Data from a Project

There are three main ways of saving data from a project::

 

 

 

See the Exporting topic for information on how to export and save data from all types of components: tables, images, drawings, maps, and other components.

 

A few words on each option:

Save the project to a .map  project file.

Using File - Save to save a project will save all data stored in that project within the .map file created for the project and it will also save all components in the project including links for linked components and connection information for data sources.    Until we save the project any edits made to local components are maintained in temporary Manifold storage.  If we quit without saving the project those changes to local components will not take effect in the project file.   Manifold will always prompt us to save changes before closing a project.

 

Note that data within linked components and data sources is always stored within the files that were linked or the data sources. When we save a project file that only saves pointers to where the data is actually stored.  

 

Saving .map project files is reasonably fault tolerant.  See the Notes section in this topic for more info.

Export a project to a .mxb  archive file.

MXB archival files are for archiving and exchanging data using the smallest possible project file size.   They are small, but slower to create or to open than a .map project file.    To export a project as an .mxb archive file, choose File - Export Project, choose MXB as the file type, provide the name desired and press Save.   Exporting to .mxb files takes significantly longer than saving a .map project file, which usually is very fast.

Export data from a component into a file.  

Right-clicking onto a component in the project pane and choosing Export in the pop up menu will allow export to those file formats Manifold supports for that type of component.    See comments in the How to Edit a Single File topic.

Export an Entire Data Source into a Project File

If we have connected to a data source such as an Oracle Server, PostgreSQL Server, ESRI geodatabase, GPKG, MySQL, SQLite or any other data source, even a web server based data source, we can harvest the entire contents of that data source and export into either a .map project file or an .mxb archival project file.   Want to capture an entire ESRI geodatabase in a single, high speed .map project file?   Easy: link the geodatabase into a project as a data source, open it to show the hierarchy within, and then right-click onto the data source, choose Export and then export as a .map or .mxb project file as preferred.  In one step we can convert a messy set of many dozens of files into a single, modern .map project that opens instantly and features massively fast and reliable storage.

Export an Automatic writes to writable linked files or data sources.  

Manifold connects to linked files and to data sources in read/write mode if possible.  As with most DBMS products, editing the data in a writable linked file or a writable data source (the default) will immediately change the saved data.  As with most DBMS products, there is no "undo" for such edits.   See the discussion in the Importing and Linking topic.

 

See also the Where Manifold Data is Stored  topic.

Use Manifold MAP Files for Storage

Manifold .map format is fast - really fast.  For larger files it is usually far faster for spatial data on almost any machine than any of the classic GIS or spatial data formats.   In many cases for spatial work it will be faster than even enterprise-class DBMS storage, even when large amounts of data are stored.

 

Manifold .map format is so fast that when building libraries of data that might be used in future projects it pays to create those libraries as collections of .map files, importing different data sets of interest from various older formats into .map files.  Doing that will allow instantaneous opens and will also allow use of "stock" data by linking those .map files into new projects with zero performance penalties.  The one-time tedium of importing a big file into Manifold .map format will be long forgotten in the many times thereafter the big file opens instantly and the data springs immediately into use.  Experience such speed once with a big file and you will not want to go back to slower formats.

 

Even better, linking a Manifold .map file into a project or connecting to that  .map file as a data source provides a means of cascading the contents of projects with zero loss of performance.    Projects can be linked one within the other to many levels without any performance loss at all.  When we link other Manifold .map files into our project we can use the contents of those files as if they were in the local file with absolutely zero delay.    That remarkable characteristic allows us to organize our data by using .map files to group data, either stored within a .map file or via links within a .map file, into logical and convenient collections.   

 

For example, we might have one project that contains one or more data sources where a complex, difficult to remember connection string is used to connect to those data sources. We can give those data sources simple, easy to remember names.   Whenever we need to use one of those data sources we can connect to the  .map file that hosts the desired data source.    

 

To take another example, we might have a collection of standardized data sets for many different countries kept as a library of .map files.   We can link in whichever one we need for a given project and performance will be as fast as if the data were imported into that project's own .map file.  See the discussion for Nested Data Sources in the Tables topic.

 

Links and data sources take up virtually no space within a project file (the data remains stored in the linked file or the data source) so we can have many project files with links and data sources in them for organizational convenience without duplicating the potentially vast amount of storage space to which they refer.   

Connecting to Manifold .map Projects

Other applications, for example GIS packages such as Manifold Release 8 or other GIS packages,  can connect to Manifold .map projects via Manifold's ODBC driver to:

 

 

 

See the examples listed in the Interoperability Examples topic for step by step instructions on how to connect to Manifold using ODBC.

Interoperability with Release 8

tech_tina_sm.png

Important: Release 9 .map format is a different, much improved .map format that is not backward compatible with Release 8, the prior generation of Manifold GIS products.  Release 9 can read old format Release 8 .map files, but Release 8 cannot read new format Release 9 .map files.   When Release 9 opens a Release 8 .map file the contents of the project are automatically converted, as much as possible, into their Release 9 equivalents.  Linking a Release 8 .map file or creating a new data source based on a Release 8 .map file will also cause that .map file to be converted into Release 9 .map file format.    Make a backup copy of any pre-Radian .map files you wish to retain in the old format before you open them or use them in a Release 9 project.

 

If we open a Release 8 .map file in Release 9, if we try to save that project Manifold will automatically increment the name so we save it to a different name, not overwriting the original Release 8 file.   This helps preserve the original Release 8 file for use with Release 8.  If we insist on writing it to the same name we can do that, but in that case the new .map file will be a Release 9 .map and not readable by Release 8.

 

To exchange data between Release 9 and Release 8 consider the following:

 

 

If access does not need to be simultaneous:

 

 

 

 

 

Notes

Manifold is a workspace for data.   It's a bigger deal than Microsoft Paint and it works with hundreds of variations of data types.   To handle all that in a regular, organized standard way we first add something to a Manifold project and then second we open it and work with it.

 

That is a different approach from programs that are designed primarily for one type of data, the way Microsoft Word is designed primarily as a word processor.  In simple programs, like Microsoft Paint, we could double-click a BMP image and have it automatically open in Paint in an editing session.

 

In contrast, if we want to just edit a single file or pop open a client session to a single database in Manifold we don't just directly "open" that file or database the way we would open a document in Word.  Instead, we first launch Manifold and add that file or database to the project and then we can work on it.    

 

Even if we only want to edit just a single .jpg image file, we do that by first launching Manifold and then we link or import the .jpg into the project as an image.   We can then open and view and edit that image, save the project with an image in it, export the image back out to a .jpg file or whatever else we want to do.   

 

The work flow of first adding data to a project and then working with it is less direct when working with just a single file but it is a lifesaver in bigger projects involving dozens, hundreds or thousands of entities.  

 

Importing bigger data into a project can be slow, possibly very slow for large files, but that only has to be done once.  After that everything goes really fast.   Once we save the imported data as a Manifold .map project it will usually be in far faster form than the original format.  Save a 100GB image or a big GIS drawing as a Manifold project and Manifold thereafter will open it instantly with performance most other environments cannot match.

 

Limitation on .map Size - Individual .map files are currently limited to 256 terabytes in size.   One terabyte is 1000 gigabytes.   Should the user community find that 256 terabyte limit too small, it can and will be increased.

 

Fault Tolerant .map Saves -  The process Manifold uses for saving a .map file is reasonably fault-tolerant, taking measures to greatly reduce the chances of .map file corruption as follows:

 

tech_lars_sm.png

 

Important: By default .map file data sources use .SAVEDATA.  If .SAVEDATA is not used and the system crashes during a save operation, literally between the writes, the resulting .map file will no longer load.  Given that the universe is designed by Fate to ensure both the main power and battery backup together will fail exactly in the middle of a critical write, it is generally a bad idea to tempt Fate by turning off the use of .SAVEDATA.   Fate always wins.

 

See Also

Getting Started

 

Importing and Linking

 

User Interface Basics

 

File - Export Project

 

Example: Convert an ESRI File Geodatabase into a .map Project

 

Example: Convert an ESRI Personal Geodatabase into a .map Project

 

Where Manifold Data is Stored

 

Accessory Files Created

 

The MAPCACHE File

 

Performance Tips

 

For Maximum Reliability