How to Edit a Single File

Most Manifold users keep all of their data in Manifold .map files.  Open the .map and edit the contents as you like.  Fast and easy!

 

We can also keep data in other formats, such as shapefiles.   We might also want to edit data within a database like PostgreSQL, GPKG or similar.   That is easy to do as well.   In most cases of popular formats we simply link them into our project using File - Link and then edit them.   Editing linked files automatically updates them when we save the project.   Linking a file into a Manifold project allows us to edit it "in place."   The data remains in the original file. See the discussion in the Importing and Linking topic.

 

Depending on the file format, we add a file to the project either by linking to the file or by importing the file's data into the project:

 

 

Link a File and Edit

Make a backup copy of the file before editing the original.  

To edit a file in place by linking:
  1. Launch Manifold.

  2. Use File - Link or File - Create - New Data Source to link the file into the project.

  3. Edit as desired.

  4. Save the project to ensure edits are saved to the file.

 

File - Link is easier for simple situations like editing a shapefile. File - Create - New Data Source provides more options and is better for connecting to sophisticated sources such as file databases or database servers.  

 

See the Example: Edit a Shapefile In Place topic showing how to edit a shapefile in place.tech_tina_sm.png

 

Please note: When using linked files changes to some file formats, like edits in tables in a linked .mdb file database, will take immediate effect in the file the moment we make them.  Edits made with some other formats might only take effect if we save the project, with the edits temporarily being done in a cached working copy of the data within Manifold and not yet saved out to the file.  While learning how various connections work we should assume all edits happen immediately and should not rely on quitting Manifold without saving as a means of abandoning edits.

 

i_linked_bronze_file.png

i_yellow_cylinder.png

Manifold shows linked data within a Project pane hierarchy that is marked with a yellow database cylinder.  The yellow database cylinder tells us all data within that hierarchy is stored somewhere outside of the project and is being linked into the project.  Linking the standard example bronze.jpg file into a project results in a Project pane display like the above.   That may seem like a lot of infrastructure just to edit a single, small file in place, but by using the same infrastructure all the time Manifold both reduces what we have to learn and is always ready to apply the full power of Manifold in every task, no matter how small.   See the discussion below regarding use of Manifold infrastructure on imports.

 

If Manifold does not allow us to edit a linked file, for example, if a table's columns have gray background indicating they are read-only and cannot be edited, that means either Manifold does not support dynamic connections to that format or that for some other reason (such as Windows security settings) the file is not writable.

 

See also the Where Manifold Data is Stored topic.

Import a File and Edit

Make a backup copy of the file before editing the original.  

To edit a file using Import and Export:
  1. Launch Manifold.

  2. Import the file's contents using File - Import.

  3. Edit those contents as desired.

  4. Export back to the file using File - Export.

  5. Close the project without saving if the imported contents are not needed.

 

Further Discussion

It may appear that the import, edit and export cycle is always an option for editing a file since Manifold can import data from very many formats.  But there are several complications in that happy picture:

 

Large Imports take time - Importing data from big files can be slow while the data is converted into the special form Manifold uses internally to support fast performance.  Manifold cares about making work within Manifold as fast as possible, since that is what Manifold users spend most of their time doing, not about making faster round trips for import, edit and export.  Once imported data gets into Manifold it can be worked with astonishing speed, almost certainly much faster than in its original environment.  It can be saved as a Manifold project that thereafter can be opened instantly and saved instantly.  But to get that speed we pay a one-time cost of the time it takes to convert the data into fast Manifold form.   If we only intend to do a one-time edit it could be faster just to leave the data in place and edit it as a linked file, if possible.

 

Linking Large Files also takes time - When linking a large file, by default Manifold will build a MAPCACHE accessory file to enable faster viewing and editing of the linked file.  That can take a long time for large files but only the first time: once a .MAPCACHE file has been built thereafter linking that same file will be very fast.  Whether it is quicker to edit a single, large file by linking or by importing is often a toss-up, the choice being made on whether it is more important to edit the file "in place" because the file may be used by other programs that expect the original format.

 

Limited export formats - Manifold can import data from many more different file formats than it can export, so it could be we might be able to import data from a file format but then not be able to export back into that file format.   See the Notes below for why that is the case.

 

Round trip translation - Importing casts data into Manifold form, the nearest analog to whatever was the original form.  Exporting data back out to the original file will cast Manifold data back into whatever is the nearest equivalent in that file format to Manifold data.   Given the possibility of differences during translation the result of the round trip might not be absolutely identical to what an "in place" edit would do.

 

Manifold infrastructure - Manifold infrastructure may manage data imported from a file using a different representation than whatever original program created that file or what we might expect, in which we might have to learn more about how to use Manifold than expect for what we hoped would be a simple in and out task.  

Round Trip Translation

The primary reason for Manifold's ability to import data from many formats is to provide many ways to get data into Manifold to do work within the very rich and fast Manifold environment. That is the main focus, not being a universal file editor for hundreds of other applications, or to export data to formats that became obsolete in the 1980's.  It is the data that interests us.  If there is some data somewhere we want, Manifold wants to make it possible for us to use that data in our projects regardless of the format in which that data comes.

 

Once data is imported into Manifold it becomes data in whatever is the Manifold data type that is the best, most analogous container to whatever type that data was in the originating format.   Importing data converts it into Manifold data within the Manifold project with no backward link whatsoever to the originating file.   Importing data takes a copy of the data from the original file and casts it into whatever is the Manifold form that is closest to the original or the best form of that original for our purposes.   

 

For example, .csv files are plain text files that typically provide one record per line with the field values for each record being text that is separated by a comma between each field value.   Texts that are intended to be string values are usually set off by double quote " characters with everything else intended to be interpreted as some sort of number.    Manifold will import a table stored as a .csv file as a table that has both string and numeric values, automatically interpreting the text representations of numbers as numeric types.

 

When we import a .csv file the data we import is cast into Manifold data types and from then on we are working in a purely Manifold world that has no connection to the world of the originating file.   That could be a subtle difference, like working with a floating point number such as 3.140000 instead of what was the text string 3.14 in an originating .csv file, but it could be a difference that will show up when we go to export our work back to the original file format after doing something with it in Manifold.  

 

When we export from Manifold to some non-Manifold file format, Manifold will do its best to take whatever is the Manifold data being exported and to cast it into the closest possible form that is consistent with the destination file format's capabilities.    The round trip of importing data from a file into Manifold form, editing it with Manifold tools and then exporting what has become purely Manifold data back into that file can easily change the data in ways other than any simple edits that might have been made.   The result should be functionally equivalent to what would have happened if the data were edited in place without an import / export round trip but it may not be identical.

 

For example, if we use Microsoft Notepad to open a .csv file and we edit the text to Replace all occurrences of the text string 3.14 with 3.1415 we will end up with a .csv text file that has that text changed with no other changes.  Consider now if we import that .csv file into Manifold.   The text in the records with the characters 3.14 will be imported as floating point numeric type so when we look at a field with that value in the resulting table we will see 3.140000.  If we edit the table to change all 3.140000 values to 3.141500 values and then export the table out to the .csv file, the result will be a text file that contains the characters 3.141500 where formerly there were the characters 3.14.  

 

Logically and from a data perspective what we have written out is, of course, a functionally equivalent version of the changes we might have made with Notepad: a floating point value of  3.141500 is indeed equivalent to the floating point value of 3.1415.   But the text representation in the .csv file as edited by Notepad "in place" will be different than the text representation in the .csv file as edited via a round trip of importing  into Manifold, editing and then exporting back out to the .csv.   The Notepad version will not have extra 00 characters at the end of each 3.1415 string.

 

The extra 00 characters in one version but not in the other version won't make a difference to any sensible program that thinks of numbers as numbers and not as text tokens, but it might make a difference to some program somewhere that chokes if it sees more than four characters following a period in what it takes to be a representation of a number.  For example, that surveying program our eccentric Uncle Bob wrote in his youth which assumed that nobody would ever want greater precision than four places past the decimal point might not like the equivalent, but not identical, edited version Manifold exported.

Manifold Infrastructure and Representations

When we import a file into Manifold we bring into Manifold a copy of the data that file contains and that data will be cast entirely into Manifold form.  Where it came from doesn't matter as once the data is in Manifold it is Manifold all the way.   In addition to data type translations such as from text strings to floating point numeric data types, the structure of how Manifold represents imported data might surprise us if it seems different than how we might work with it in a simpler editor.   It may be logically equivalent and a better way of representing the data for Manifold purposes, but it may still surprise someone new to Manifold.   

 

The classic example is importing a .jpg image into Manifold.  When we open a .jpg image in almost any image editor, such as Adobe Photoshop, we usually just see an image pop open.   

 

i_imported_bronze_file.png

 

In contrast, when we import a .jpg image into Manifold, we see that both an image and a table are created are created in the Project pane.  The table which also is created is a key part of Manifold infrastructure that makes very many capabilities possible, but it might confuse us the first time we see it.   Having a table in addition to an image is not a big deal for most image editing operations as we can simply work with the image, edit it and then export it.  If we are not doing more sophisticated things we might not consider, or even know, that the data for the image is actually stored in the associated table, the image just being a way to view the raster, tile data in the table.   But seeing a table appear in addition to the image component in the Project pane might be a surprise for beginners.

 

Besides the notion of treating even very small images as data in a table there are other Manifold infrastructure details that may seem unnecessarily intricate for the simple task of, say, opening a single image and editing it.   For example, Manifold's use of x-y-tile indexes or level-x-y indexes, the generation of intermediate levels and so on may seem like overkill when working with a small .jpg.   But those details are all part of the infrastructure that allows Manifold users to work with breathtakingly large images, to analyze such images using massively parallel functions and to take advantage of the sophisticated armature of SQL and big-time DBMS thinking within extensive constellations of different types of data, which also happen to include images.   Those are all good things with which a simple image editor like Microsoft Paint cannot help us.   

 

The point of Manifold, after all, is not to clone Microsoft Paint.  It is to provide a package that can instantly pop open a project with 50 GB images and work with sophisticated image data that can involve many channels of sophisticated numeric types, and to do that within the context of what could be thousands of other significant collections of complex data.  Given that Manifold users will learn how to use those capabilities in any event, it is more straightforward and no harm done to use the same infrastructure and user interface even for very simple tasks involving small data.

Notes

Data imported into Manifold is purely Manifold - Once we import data into Manifold all connection to where that data came from and is no more. "Importing a file" is an summary way of saying "make a copy of what is in the file and bring that into Manifold, converting on the fly into Manifold stuff."  Imported data becomes converted into some Manifold data type stored within one or more Manifold components and should be discussed in exclusively Manifold terms.  

 

For example, it is a beginner's mistake to say "I imported my JPEG image into Manifold, and then I opened the JPEG.  How do I edit the JPEG?" Once the data from a .jpg image has been imported into Manifold the image we see in the Manifold project is no longer a "JPEG" image.   It is a Manifold image of some sort, probably made up of 128 x 128 sized tiles where each tile is an array of unint8x3 type values, the Manifold data type used to represent RGB values by default.    Experienced Manifold users are not going to say "hey, I just opened my unint8x3 tiled image..." but they won't call it a "JPEG" either.  They'll just call it an image.

 

Why not an Export for every possible Import?  It may sound odd at first that Manifold can import from a file format but not export to that format.  One reason that happens is that many file formats are either very old or very limited and they cannot handle the contents we may create or other results of editing we do with a rich tool in modern times.  The goal of truncating rich, modern usage into lesser, limited forms sometimes is not worth pursuing.

 

For example, the file database format .dbf dates back to dBase II in the late 1970's.   It has extreme limitations on the names of fields and the contents of fields that are highly incompatible with modern workflow and user preferences.  Data can easily enough be imported from such old formats because it is in simple, limited form.  Given that there is a lot of data archived in old formats having the ability to import from historic formats is very useful.   

 

But when that imported data is then edited in a robust package like Manifold it can easily be transformed into something that no longer will fit into the original format.  Do something as simple as changing the name of a field from IN90DAT to 1990 Income  to make it more comprehensible and suddenly the table is no longer compatible with .dbf field naming limitations.    An expert in .dbf might remember all the arcane limitations on no spaces in field names, length limits, limits on not starting with number and so on, but for most people when they edit data using modern tools it is unrealistic to expect they will be experts in the limitations of a format from many decades past.   

 

The Manifold exporter, then, would have to take on the task of automatically dumbing down modern, rich content into the much more limited content allowed by a highly limited format like .dbf.   That's simply not worth doing in modern times, at least not for all possible formats if our priority is putting our resources and our best efforts into utilizing modern technology to get work done faster and better.  Data interchange between mutually competent programs is a part of modern workflow, of course, but the most effective way to serve such data interchange is to choose a limited set of formats that are most frequently used for data interchange and to provide exporters for those formats.

 

Why not a simple click open a file?  Naturally if we are used to a one step process of opening an image with an image editor or a document with Word we might wonder why Manifold requires a two step process of first adding to a project (either by linking a file or importing from it) and then editing.

 

That is an effect of using an application designed to make more complex workflow as easy as possible and not to make the simple workflow as easy as possible.  The two are different things.    There's nothing wrong with using a more sophisticated tool to do simple things if that is what we need to do but we should consider that the more sophisticated tool will not benefit from the simplicity that can be achieved by aiming primarily at simpler tasks.

 

Consider that Manifold's mission in life is to enable sophisticated work with large data sets, especially those data sets that have meaning based on location and those data sets that involve significant amounts of data within each record.   That mission drives what we see in Manifold infrastructure and how Manifold presents a working environment which makes such complex work easier to accomplish.

 

Manifold is geared up to connect to enterprise-class databases that can involve thousands of tables and other structures or to maintain within the Manifold project similarly sophisticated constellations of tables, queries, scripts, images, drawings and other components that could easily involve hundreds of gigabytes of data.  A Manifold project can do both, maintaining hundreds of gigabytes within the Manifold project while simultaneously connecting to terabytes of data in external databases.  

 

That is why the Manifold interface uses the Project Pane to present what could be thousands of components stored in the project or linked into the project from big, sophisticated, external databases.  Manifold expects we will want to work with many different components from many different sources at the same time so it starts by getting them into the Project pane, onto the Manifold desktop where we can work with them.   The interface is weighted towards the assumption we will be working with many tables, images, drawings or whatever and not just one.

 

That is a somewhat different approach from an editor or client that has one job in life, to open and to edit a particular type of file that contains more or less one type of well-known data, and to do that one job starts with the assumption that the file should be opened, in place, for editing.   We can, of course, use Manifold to open a particular type of file, edit it and then save the results, doing that either on-the-fly with the data in place  in the case of linked file or we an do it in an import, edit and export cycle that begins by first bringing data into Manifold.   But in both cases taking advantage of Manifold infrastructure designed to make the most complicated things more manageable means that very simple things may require an extra step.

 

See Also

Getting Started

 

The MAPCACHE File

 

Tables

 

Drawings

 

Images

 

Where Manifold Data is Stored

 

Performance Tips

 

Example: Edit a Shapefile In Place - How to edit a shapefile "in place," that is, leaving the data in the shapefile and only linking it into a project and not importing it into the project.