Projects and .map Files

Manifold project files, called projects, in .map format are the native "documents" for Manifold.   The File - Open command will open an existing Manifold .map project file and the File - Save command will save a Manifold project in .map project file format.  Manifold .map format is also referred to as MAP format, using all-capital letters.

 

Manifold stores everything it does in a project within a Manifold .map file.    A .map project file is an incredibly fast, very rich, file database that can store data of all kinds.  It can store large, complex entities like maps, drawings, scripts, queries, tables and much more, and it can store links to other databases, like Oracle or PostgreSQL DBMS servers, and it can store connections to web data sources providing terabytes of data.   A .map project file can have thousands of entities in it and itself be hundreds of gigabytes in size.  

 

Manifold .map project format is an operational format: it is designed for everyday work.  Manifold .map format is built for maximum speed, and for nesting of projects with zero performance loss.   .map project format is incredibly fast, convenient, and very robust, a perfect format for daily work.

 

Manifold also has another format, .mxb format, that is designed to store projects as compactly as possible for archival storage and for exchanging data.    .mxb format is also referred to as MXB format, using all-capital letters.  MXB is slower to write and slower to open than MAP and is cannot be used operationally for nesting projects.   We can open a .mxb file with Manifold, and Manifold will immediately decompress it to create a .map format project that it will use.  MXB is a highly compressed and robust format that minimizes file size, perfect for archival storage and for minimizing the size of files for transmission.   MXB is also useful in ensuring maximum data reliability.  See the For Maximum Reliability topic.

 

Manifold .map files at the present time do not allow simultaneous read/write access by multiple Manifold sessions.   If a .map file has been opened in one Manifold session it cannot be opened in another.  We can, however, link a .map file into several different Manifold sessions if we create the data source for each with the read only check box checked.

 

If we require simultaneous read/write access to data by multiple Manifold sessions, such as by different people on different machines, we can store the data in a DBMS, such as PostgreSQL/PostGIS, Oracle, MySQL, SQL Server, DB2, or other DBMS.

 

Always run the latest version of Release 9 or Viewer.   If you cannot open a Release 9 .map or .mxb project file, install the latest version of 9 or Viewer and try again.  Older versions of 9 or Viewer might not be able to open projects created by the very latest build.

Projects

A project is the collection of stuff we are working on in any given Manifold session.   It is all the different things we have in our Manifold workspace in that session.   The Project pane to the right of the Manifold desktop will show the contents of the project.  See the View - Panes - Project topic.

 

 

 

 

 

Manifold always launches with a project, a new blank project called New Project if not opened with an existing project file.    A new project is empty but for a System Data folder containing tables Manifold uses to manage the project.

 

Clicking the + icon opens the System Data folder to show the two special tables it contains.   

 

We do not need to work with the System Data tables during routine Manifold use.   However, since they are small, simple tables and they always are at hand we can use them to illustrate examples in this documentation.

 

 

In the illustration above we have used File - Open to open a previously-saved project called Aus Hydro.map.    It all comes back, including a data source that is a Microsoft Bing image server that is ready to go.

 

Tech tip: Within the Project pane most Windows moves we know, like Copy and Paste, will work.  For example, we can Copy a table and Paste it within a folder.  We can drag and drop a table into a folder to move it there.  One modification to a standard Windows key: drag and drop in Windows is a move command and Ctrl-drag and drop is a copy command.   In Manifold, drag and drop also is a move command but Shift-drag and drop is a copy command.

Folders

We can organize what is in our project using folders, like the System Data folder seen above, a good idea if our projects include many components.   The Project pane itself includes useful features, like filtering, to help us zero in on the components we want in a project that may contain very many.

Components

When we add stuff to our project that stuff will appear in the Project pane as various items, called components. Components are items such as tables, images, scripts, data sources and so on depending on what sort of data is added.    The project pane will show all components in our project with an icon next to each to indicate the type of component.

 

 

Tables are the center of everything in Manifold.   All data in Manifold is stored in tables in one form or another.  That's good because it means we can get at anything with SQL.   Nothing is hidden.  

 

Other components such as images or drawings are visualizations of data stored within some table.  Map components show other visual components, such as images, drawings and labels, as layers within a map window.

 

Tables - Where all data within Manifold is stored in one form or another.  Opening a table window shows a table in classic row and column presentation.

Drawings - A visual display of vector data stored within tables using a variety of geometry types. The same data can be seen in multiple different drawings using different styles, and the same drawing can be open in multiple windows with different selections and views in each.

Data Sources - Data linked into a project from an external source, which could be a file, a file database, a database serve or a web server, covering virtually every database and file type encountered in database or GIS work.

Images - A visual display of raster data stored within tables as tiles using a variety of different data types and channel combinations.   All rasters are called images in Manifold even those which are terrain elevation or other non-photographic data.

Labels - Text annotations automatically created from the fields of a drawing's table.

Layouts - Layouts are compositions on virtual sheets of paper, allowing us to create documents showing maps or other components as PDF documents or hard copy printed sheets.

Locations - Locations save a particular geographic location with an associated scale, thus saving a given view in a map or other window to which we can return.

Maps - Consist of a stack of layers where each layer is a drawing, image or labels component.

Queries - Written in SQL, queries manipulate data and projects, for example, creating new tables and other components, performing calculations,  altering the structure of databases and tables, and extracting, editing and analyzing subsets of data.

Scripts - Manifold includes built-in support for scripting in eleven different languages with six to eight always available and the remainder easy to install.  Scripts provide custom capabilities and can automate virtually anything.

Comments -  Text saved and displayed in a comments window provides a simple but  convenient way to save notes about a project or to save text as a scratchpad.

Folders - We can organize our project by using folders.

Linked Components and Local Components

Components we see in the Project pane may be a link to data stored somewhere else or they may be a local component for which the data is stored within the project file.   Linked components don't take up any space in the .map file that stores the project.   They're just pointers to the file or data source somewhere else where the data is stored.   Local components are stored in the project's .map file, which will grow in size as we add local components to a project.   .map files can get very big if we add large components to them.   

 

 

A yellow database cylinder marks a linked file or data source in the Project pane.  The yellow database cylinder tells us that everything within that hierarchy is coming into the project from some source outside of the project.  The source will also have its own System Data folder with information on the linked source.   In the above example the Products table seen under the products hierarchy is coming in from some data source outside of the project.   In contrast, the bronze image and the bronze Tiles table are both local components that are stored within the project file.

 

When we use File - Save to save a project we write that project into a file database stored in Manifold .map file format.  The .map file we create saves all the data for local components as well as saving links, Project pane folder hierarchies and other housekeeping information for the project.  The next time we open the project file in Manifold we will get all of that back, with the Project Pane showing components and folders organized exactly like they were when we saved the project.

 

Manifold project files can save large amounts of data, hundreds of gigabytes, and provide performance second to none in database applications.  That makes Manifold itself a fine choice for a file database system, using Manifold .map format project files for storage, without being forced to use an external DBMS just because the tables, images, drawings or other data we would like to store and to work with are very large.

 

At the same time Manifold is exceptionally adept at working with and very fast at connecting to DBMS servers like Oracle, PostgreSQL, DB2 or SQL Server, so fast that Manifold will take full advantage of the best performance those systems can provide.   That makes Manifold a great console for working with multiple different DBMS databases at once, with big data stored locally in the project file, or with big data stored in external DBMS servers or both at the same time.

 

Manifold can also link other, previously created Manifold project files into a project with no loss of performance compared to importing the data into local storage within the project.  For example, the products link seen in the illustration above is a Manifold .map project file called products that was linked into the project.   Manifold can then work with such linked-in projects with zero loss of performance no matter how big the linked-in project may be, just as if the contents of those linked-in projects were saved locally within the session's own project file.   Working with linked project files is so fast that we would never guess they were linked files from performance alone if we did not see in the project file they were marked with a yellow database cylinder.

Getting Data into a Project

There are many ways to get data into project:

 

Open an existing Manifold .map project file or Manifold .mxb archive file.    

Use File - Open to open a Manifold .map project file or a Manifold .mxb archive file.  

 

 

 

 

Large .map files in pre-Radian format can take a long time to be converted into Radian .map file format.  Be patient in such cases as once the project is saved by Manifold as a Radian .map file it will thereafter open instantly. 

 

Create new components.   

We can use the File - Create menu (or right-click into the Project pane and choose from the context menu) to create a new component such as a table or a drawing and then add data manually  to that component, or add data programmatically, taking data from some process that generates data.   The simplest way to add data manually is to create a table, specifying what fields we want in the table and to then proceed to add records one by one manually to the table.  We are not going to manually add a billion records to a table but a few hundred are easy to add and some organizations over time manually  will create tables with thousands of records.  

 

We can also create a drawing and then manually create objects in that drawing. Manifold System provides exceptionally rich editing capabilities for drawings.  The DBMS-centric subset of Manifold, Radian Studio, is not a GIS or CAD editor so editing facilities within Radian Studio for drawings are limited.   But for many purposes even the limited capabilities within Manifold are more than enough to allow viewing of drawings and the creation of new objects, for example, by tracing over what is seen in an imageserver.

Copy and Paste between Manifold sessions.  

If we launch more than one Manifold session we can open different projects with those sessions and then Copy and Paste between projects.   Because Manifold projects open instantly they make very good storage containers for collections of data.   Users will often have terabytes of folders on disk that are full of Manifold projects that contain their data holdings.   The same .map format is used for all Radian-based applications, so Manifold System can open and work with .map projects created and saved by Radian Studio.

 

We can use Manifold projects by linking them into our project or by popping the project open in a separate Manifold session and then copying and pasting from it to our current session. The best way to get data from one Manifold project into another is to create a data source for that other project within the session where we are working.   That is generally faster than making copies using the Windows Clipboard.   

 

Tech tip: When copying records from a table and pasting into another table, to add records to the destination table we should use the Ctrl-Shift-V command for a  Paste Append.   A paste append will automatically avoid collisions between existing and pasted values of the mfd_id key field used in tables that contain data for drawings and images because the pasted records will automatically be given new, unique mfd_id values.  If we neglect to use Ctrl-Shift-V we might not be able to paste any records because mfd_id values must be unique: if the records we would like to paste have any mfd_id values that already exit in the destination table they will not be pasted into that table.

Write scripts and programs to create new data or to acquire it.

We can utilize a variety of automated means to create new components and to populate them with data.   We can use SQL to create new tables and other components, either from existing ones or as a result of computations.   We can use SQL or scripting or a combination to connect to external sources, for example, external sensors such as GPS devices or cameras which generate data, and to capture that data within our project.

Saving Data from a Project

There are three main ways of saving data from a project::

 

 

 

See the Exporting topic for information on how to export and save data from all types of components: tables, images, drawings, maps, and other components.

 

A few words on each option:

Save the project to a .map  project file.

Using File - Save to save a project will save all data stored in that project within the .map file created for the project and it will also save all components in the project including links for linked components and connection information for data sources.    Until we save the project any edits made to local components are maintained in temporary Manifold storage.  If we quit without saving the project those changes to local components will not take effect in the project file.   Manifold will always prompt us to save changes before closing a project.

 

Note that data within linked components and data sources is always stored within the files that were linked or the data sources. When we save a project file that only saves pointers to where the data is actually stored.  

 

Saving .map project files is reasonably fault tolerant.  See the Notes section in this topic for more info.

Export a project to a .mxb  archive file.

MXB archival files are for archiving and exchanging data using the smallest possible project file size.   They are small, but slower to create or to open than a .map project file.    To export a project as an .mxb archive file, choose File - Export Project, choose MXB as the file type, provide the name desired and press Save.   Exporting to .mxb files takes significantly longer than saving a .map project file, which usually is very fast.

Export data from a component into a file.  

Right-clicking onto a component in the project pane and choosing Export in the pop up menu will allow export to those file formats Manifold supports for that type of component.    See comments in the How to Edit a Single File topic.

Export an Entire Data Source into a Project File

If we have connected to a data source such as an Oracle Server, PostgreSQL Server, ESRI geodatabase, GPKG, MySQL, SQLite or any other data source, even a web server based data source, we can harvest the entire contents of that data source and export into either a .map project file or an .mxb archival project file.   Want to capture an entire ESRI geodatabase in a single, high speed .map project file?   Easy: link the geodatabase into a project as a data source, open it to show the hierarchy within, and then right-click onto the data source, choose Export and then export as a .map or .mxb project file as preferred.  In one step we can convert a messy set of many dozens of files into a single, modern .map project that opens instantly and features massively fast and reliable storage.

Export an Automatic writes to writable linked files or data sources.  

Manifold connects to linked files and to data sources in read/write mode if possible.  As with most DBMS products, editing the data in a writable linked file or a writable data source (the default) will immediately change the saved data.  As with most DBMS products, there is no "undo" for such edits.   See the discussion in the Importing and Linking topic.

 

See also the Where Manifold Data is Stored  topic.

Use Manifold .map Files for Storage

Manifold .map format is fast - really fast.  For larger files it is usually far faster for spatial data on almost any machine than any of the classic GIS or spatial data formats.   In many cases for spatial work it will be faster than even enterprise-class DBMS storage, even when large amounts of data are stored.

 

Manifold .map format is so fast that when building libraries of data that might be used in future projects it pays to create those libraries as collections of .map files, importing different data sets of interest from various older formats into .map files.  Doing that will allow instantaneous opens and will also allow use of "stock" data by linking those .map files into new projects with zero performance penalties.  The one-time tedium of importing a big file into Manifold .map format will be long forgotten in the many times thereafter the big file opens instantly and the data springs immediately into use.  Experience such speed once with a big file and you will not want to go back to slower formats.

 

Even better, linking a Manifold .map file into a project or connecting to that  .map file as a data source provides a means of cascading the contents of projects with zero loss of performance.    Projects can be linked one within the other to many levels without any performance loss at all.  When we link other Manifold .map files into our project we can use the contents of those files as if they were in the local file with absolutely zero delay.    That remarkable characteristic allows us to organize our data by using .map files to group data, either stored within a .map file or via links within a .map file, into logical and convenient collections.   

 

For example, we might have one project that contains one or more data sources where a complex, difficult to remember connection string is used to connect to those data sources. We can give those data sources simple, easy to remember names.   Whenever we need to use one of those data sources we can connect to the  .map file that hosts the desired data source.    

 

To take another example, we might have a collection of standardized data sets for many different countries kept as a library of .map files.   We can link in whichever one we need for a given project and performance will be as fast as if the data were imported into that project's own .map file.  See the discussion for Nested Data Sources in the Tables topic.

 

Links and data sources take up virtually no space within a project file (the data remains stored in the linked file or the data source) so we can have many project files with links and data sources in them for organizational convenience without duplicating the potentially vast amount of storage space to which they refer.   

Connecting to Manifold .map Projects

Other applications, for example GIS packages such as Manifold Release 8 or other GIS packages,  can connect to Manifold .map projects via Manifold's ODBC driver to:

 

 

 

See the examples listed in the Interoperability Examples topic for step by step instructions on how to connect to Manifold using ODBC.

Interoperability with Release 8

Important: Release 9 .map format is a different, much improved .map format that is not backward compatible with Release 8, the prior generation of Manifold GIS products.  Release 9 can read old format Release 8 .map files, but Release 8 cannot read new format Release 9 .map files.   When Release 9 opens a Release 8 .map file the contents of the project are automatically converted, as much as possible, into their Release 9 equivalents.  Linking a Release 8 .map file or creating a new data source based on a Release 8 .map file will also cause that .map file to be converted into Release 9 .map file format.    Make a backup copy of any pre-Radian .map files you wish to retain in the old format before you open them or use them in a Release 9 project.

 

If we open a Release 8 .map file in Release 9, if we try to save that project Manifold will automatically increment the name so we save it to a different name, not overwriting the original Release 8 file.   This helps preserve the original Release 8 file for use with Release 8.  If we insist on writing it to the same name we can do that, but in that case the new .map file will be a Release 9 .map and not readable by Release 8.

 

To exchange data between Release 9 and Release 8 consider the following:

 

 

If access does not need to be simultaneous:

 

 

 

 

 

Limitations

In most cases, we will run out of machine resources before hitting any limitations of Manifold.  There are a few limitations that may be encountered even in large machines:

 

256 GB max size for .map files - Using 64-bit Manifold, the amount of data stored within a .map file cannot be larger than 256 GB, although a .map file can contain data sources linking to a DBMS that contain far larger data, petabytes, in size.   The maximum .map file size is 64 GB with 32-bit Manifold.   The 64-bit limitation will be increased in future builds.

 

256 GB max size for changes to data - .map files can work with data far larger than 256 GB when that data is stored in external data sources, such as an enterprise class DBMS.  Although there is no theoretical limit to the size of data Manifold can work with on the desktop, a hard limit on the amount of data that can be changed in a single operation is 256 GB.   For example, if we have a single image that is 1 terabyte in size we can display that and manipulate it, but if we want to change all of the values in all of the pixels in the image we will have to do that in four operations, each up to 256 GB in size.   The 256 GB limit is the current, somewhat arbitrary, setting for temporary caches utilized internally by Manifold.   This limitation will be increased in future builds.

 

See the Limitations topic for a current list of limitations.

Notes

Manifold is a workspace for data.   It's a bigger deal than Microsoft Paint and it works with hundreds of variations of data types.   To handle all that in a regular, organized standard way we first add something to a Manifold project and then second we open it and work with it.

 

That is a different approach from programs that are designed primarily for one type of data, the way Microsoft Word is designed primarily as a word processor.  In simple programs, like Microsoft Paint, we could double-click a BMP image and have it automatically open in Paint in an editing session.

 

In contrast, if we want to just edit a single file or pop open a client session to a single database in Manifold we don't just directly "open" that file or database the way we would open a document in Word.  Instead, we first launch Manifold and add that file or database to the project and then we can work on it.    

 

Even if we only want to edit just a single .jpg image file, we do that by first launching Manifold and then we link or import the .jpg into the project as an image.   We can then open and view and edit that image, save the project with an image in it, export the image back out to a .jpg file or whatever else we want to do.   

 

The work flow of first adding data to a project and then working with it is less direct when working with just a single file but it is a lifesaver in bigger projects involving dozens, hundreds or thousands of entities.  

 

Importing bigger data into a project can be slow, possibly very slow for large files, but that only has to be done once.  After that everything goes really fast.   Once we save the imported data as a Manifold .map project it will usually be in far faster form than the original format.  Save a 100GB image or a big GIS drawing as a Manifold project and Manifold thereafter will open it instantly with performance most other environments cannot match.

 

Fault Tolerant .map Saves -  The process Manifold uses for saving a .map file is reasonably fault-tolerant, taking measures to greatly reduce the chances of .map file corruption as follows:

 

 

Important: By default .map file data sources use .SAVEDATA.  If .SAVEDATA is not used and the system crashes during a save operation, literally between the writes, the resulting .map file will no longer load.  Given that the universe is designed by Fate to ensure both the main power and battery backup together will fail exactly in the middle of a critical write, it is generally a bad idea to tempt Fate by turning off the use of .SAVEDATA.   Fate always wins.

 

Open a .map file from a command prompt -  For users who want to use a batch file to launch Manifold or Viewer with a particular .map project file, Manifold supports the usual Windows way of opening a file as a command line argument for the opening program:  Running manifold.exe myproject.map will launch Manifold and open myproject.map in Manifold.  Use double quotes around the file name if there are any special characters or spaces in the .map file name.  For example, we would write manifold.exe "-this is my latest project.map".

 

See Also

Getting Started

 

Importing and Linking

 

User Interface Basics

 

File - Export Project

 

Example: Convert an ESRI File Geodatabase into a .map Project

 

Example: Convert an ESRI Personal Geodatabase into a .map Project

 

Where Manifold Data is Stored

 

Accessory Files Created

 

The MAPCACHE File

 

Performance Tips

 

For Maximum Reliability