SQL Example: Process RGB Images using Matrix Filters

This topic is a continuation of the SQL Example: Process Images with 3x3 Filters  topic and the SQL Example: Process Images using Dual 3x3 Filters topic. Please read those two topics in order before continuing, as this topic picks up directly where that last topic finished.   

 

The discussion in this topic is the third in a series of three topics, beginning with the SQL Example: Process Images with 3x3 Filters  topic and continuing the SQL Example: Process Images using Dual 3x3 Filters topic.

RGB Images

So far, our example query has operated only on single-channel images, displayed as grayscale images like the st_peters_grayscale image below.

 

eg_process_images_3x3_stpeters_grayscale.png

 

We would like to extend the example query so it applies convolution matrix processing to three-channel, RGB images like the st_peters image below.  

 

eg_process_images_3x3_stpeters.png

 

A single-channel image has tiles that for each pixel contain a single number, usually a integer such as a UINT8 or a float such as a FLOAT64.   A three-channel, RGB image has tiles that for each pixel contain a vector consisting of three numbers, such as a UINT8x3 or a FLOAT64x3.     In the case of processing a single-channel image with a filter matrix the computation happens on the single number for each pixel.   We can apply an analogous procedure to RGB images by applying that single-channel computation to each of the three numbers in turn within the RGB pixel.

 

Our compartmentalization of processing within the processtile function will make it easier to write a multi-channel version of the query.   We will also introduce a new processRGB function to modularize some of the extra work that needs to be done to split out channels for processing and then reassembly of those channels.   We first will create a single filter example, using a 5x5 Unsharp matrix, and then we will create a two-filter, Prewitt version.

Splitting a Three-Channel Tile into Three,  Single-Channel Tiles

We can learn how to write SQL that takes a three-channel, RGB image like the one above and creates a single-channel image.   To learn how to do that, we turn to our old friend and SQL tutor, the Edit Query button.

 

With the focus on an RGB image, such as the st_peters image shown above, we launch the Transform pane.  We choose Tile as the target field and the Channel template.   We leave the Channel value set to channel 0 since we do not care which channel is extracted.  We simply want to see how it is done.  

 

eg_process_images_3x3_18.png

 

We press the Edit Query button to see what SQL Manifold uses to pull one channel out of a three-channel RGB image.   The resulting query:

 

-- $manifold$

--

-- Auto-generated

-- Transform - Channel - Add Component

--

CREATE TABLE [st_peters Tiles Channel] (

  [X] INT32,

  [Y] INT32,

  [Tile] TILE,

  [mfd_id] INT64,

  INDEX [mfd_id_x] BTREE ([mfd_id]),

  INDEX [Y_X_x] BTREE ([Y], [X]),

  INDEX [X_Y_Tile_x] RTREE ([X], [Y], [Tile] TILESIZE (128, 128) TILETYPE UINT8),

  PROPERTY 'FieldTileSize.Tile' '[ 128, 128 ]',

  PROPERTY 'FieldTileType.Tile' 'uint8'

);

CREATE IMAGE [st_peters Tiles Channel Image] (

  PROPERTY 'Table' '[st_peters Tiles Channel]',

  PROPERTY 'FieldTile' 'Tile',

  PROPERTY 'FieldX' 'X',

  PROPERTY 'FieldY' 'Y',

  PROPERTY 'Rect' '[ 0, 0, 1214, 862 ]'

);

PRAGMA ('progress.percentnext' = '100');

INSERT INTO [st_peters Tiles Channel] (

  [X], [Y],

  [Tile]

) SELECT

  [X], [Y],

  CASTV ((TileChannel([Tile], 0)) AS UINT8)

FROM [st_peters]

THREADS SystemCpuCount();

TABLE CALL TileUpdatePyramids([st_peters Tiles Channel Image]);

 

Setting aside the infrastructure of creating a destination table and image and then populating it, all of the action happens using a single invocation of the TileChannel function:

 

  CASTV ((TileChannel([Tile], 0)) AS UINT8)

 

From the function's description in the SQL Functions topic, we know TileChannel takes two arguments, a tile with multiple channels and a number for which channel is desired (counting starting with zero), and then it returns a single-channel tile containing the extracted channel.   

 

Now that we know how to pull a single channel out of a three-channel RGB image, we can proceed to adapting our single-channel query to process in turn each of the single channels we will extract.  We then will put them back together into a three-channel RGB tile.   To learn how to do that, we read the Example: Import BIL and Combine 3 Bands topic, which shows how to combine three, single-channel tiles into a single three-channel tile.  

 

In that example, if [Tile1], [Tile2], and [Tile3] are each single-channel tiles, we can combine them using the TileChannelConcat function:

 

TileChannelsConcat([Tile1],

TileChannelsConcat([Tile2], [Tile3]))

 

The above expression returns a three-channel Tile that is formed by concatenating three single-channel tiles.

 

We now have all of the pieces we need to adapt the single-channel example queries into three-channel RGB queries.

Adapting a Single-Channel Query

Consider the single-channel 5x5 Unsharp filter query we saw at the end of the SQL Example: Process Images with 3x3 Filters  topic, adapted to use the st_peters_grayscale image:

 

-- $manifold$

--

 

VALUE @source_image TABLE = [st_peters_grayscale];

VALUE @source_image_rect  NVARCHAR = '[ 0, 0, 1214, 862 ]';

 

-- Unsharp 5x5

VALUE @filter TILE = StringJsonTile('[

-0.00391, -0.01563, -0.02344, -0.01563, -0.00391,

-0.01563, -0.06250, -0.09375, -0.06250, -0.01563,

-0.02344, -0.09375,  1.85938, -0.09375, -0.02344,

-0.01563, -0.06250, -0.09375, -0.06250, -0.01563,

-0.00391, -0.01563, -0.02344, -0.01563, -0.00391

]', 5, 5, 1, true);

 

VALUE @radius UINT8 = 2;

 

CREATE TABLE [Custom] (

  [X] INT32,

  [Y] INT32,

  [Tile] TILE,

  [mfd_id] INT64,

  INDEX [mfd_id_x] BTREE ([mfd_id]),

  INDEX [X_Y_Tile_x] RTREE ([X], [Y], [Tile] TILESIZE (128,128) TILETYPE FLOAT64),

  PROPERTY 'FieldTileSize.Tile' '[ 128, 128 ]',

  PROPERTY 'FieldTileType.Tile' 'float64'

);

CREATE IMAGE [Custom Image] (

  PROPERTY 'Table' '[Custom]',

  PROPERTY 'FieldTile' 'Tile',

  PROPERTY 'FieldX' 'X',

  PROPERTY 'FieldY' 'Y',

  PROPERTY 'Rect' @source_image_rect

);

 

FUNCTION processtile(@t TILE, @r UINT8, @f TILE) TILE AS (

  TileFilter(@t, @r, @f)

) END;

 

PRAGMA ('progress.percentnext' = '100');

INSERT INTO [Custom] (

  [X], [Y],

  [Tile]

) SELECT

  [X], [Y],

  CASTV ((TileRemoveBorder(processtile(

    TileCutBorder(@source_image, VectorMakeX2([X], [Y]), @radius), @radius, @filter

    ), @radius)) AS FLOAT64)

FROM @source_image

THREADS SystemCpuCount();

ALTER TABLE [Custom] (

  ADD PROPERTY 'FieldCoordSystem.Tile' ComponentCoordSystem(@source_image)

);

TABLE CALL TileUpdatePyramids([Custom Image]);

 

The heart of the query is what happens in the processtile function, where TileFilter does its work:

 

FUNCTION processtile(@t TILE, @r UINT8, @f TILE) TILE AS (

  TileFilter(@t, @r, @f)

) END;

 

It seems that to adapt the query to RGB, all we need do would be to apply the TileFilter function to each of the three channels within the RGB image.   To do that, we will use the TileChannel function to extract each of the three channels from the three-channel RGB tile.   We will then process the single-channel tiles.   Finally, we will put the results tiles back together into a three-channel tile using TileChannelsConcat.

 

To keep our code more modular, we will introduce a new function, a processRGB function, to host all the salad dressing involved in using TileChannel three times and then concatenating the results with TileChannelsConcat:

 

FUNCTION processRGB(@t TILE, @r UINT8, @f TILE) TILE AS (

TileChannelsConcat(

  processtile(TileChannel(@t,0), @r, @f),

TileChannelsConcat(

  processtile(TileChannel(@t,1), @r, @f),

  processtile(TileChannel(@t,2), @r, @f)))

) END;

 

The function breaks out each channel, applies processtile, and then concatenates that result into a three-channel tile.

First Version: RGB 5x5 Unsharp Filter

We can now rewrite our single-channel query for use with RGB images:

 

-- $manifold$

--

 

VALUE @source_image TABLE = [st_peters];

VALUE @source_image_rect  NVARCHAR = '[ 0, 0, 1214, 862 ]';

 

-- Unsharp 5x5

VALUE @filter TILE = StringJsonTile('[

-0.00391, -0.01563, -0.02344, -0.01563, -0.00391,

-0.01563, -0.06250, -0.09375, -0.06250, -0.01563,

-0.02344, -0.09375,  1.85938, -0.09375, -0.02344,

-0.01563, -0.06250, -0.09375, -0.06250, -0.01563,

-0.00391, -0.01563, -0.02344, -0.01563, -0.00391

]', 5, 5, 1, true);

 

VALUE @radius UINT8 = 2;

 

CREATE TABLE [Custom] (

  [X] INT32,

  [Y] INT32,

  [Tile] TILE,

  [mfd_id] INT64,

  INDEX [mfd_id_x] BTREE ([mfd_id]),

  INDEX [X_Y_Tile_x] RTREE ([X], [Y], [Tile] TILESIZE (128,128) TILETYPE FLOAT64x3),

  PROPERTY 'FieldTileSize.Tile' '[ 128, 128 ]',

  PROPERTY 'FieldTileType.Tile' 'float64x3'

);

CREATE IMAGE [Custom Image] (

  PROPERTY 'Table' '[Custom]',

  PROPERTY 'FieldTile' 'Tile',

  PROPERTY 'FieldX' 'X',

  PROPERTY 'FieldY' 'Y',

  PROPERTY 'Rect' @source_image_rect

);

 

FUNCTION processtile(@t TILE, @r UINT8, @f TILE) TILE AS (

  TileFilter(@t, @r, @f)

) END;

 

FUNCTION processRGB(@t TILE, @r UINT8, @f TILE) TILE AS (

TileChannelsConcat(

  processtile(TileChannel(@t,0), @r, @f),

TileChannelsConcat(

  processtile(TileChannel(@t,1), @r, @f),

  processtile(TileChannel(@t,2), @r, @f)))

) END;

 

PRAGMA ('progress.percentnext' = '100');

INSERT INTO [Custom] (

  [X], [Y],

  [Tile]

) SELECT

  [X], [Y],

  CASTV ((TileRemoveBorder(processRGB(

    TileCutBorder(@source_image, VectorMakeX2([X], [Y]), @radius), @radius, @filter

    ), @radius)) AS FLOAT64)

FROM @source_image

THREADS SystemCpuCount();

ALTER TABLE [Custom] (

  ADD PROPERTY 'FieldCoordSystem.Tile' ComponentCoordSystem(@source_image)

);

TABLE CALL TileUpdatePyramids([Custom Image]);

 

As can be seen above, the modifications (indicated in magenta) are remarkably few, thanks to the modularization of code we achieve by using functions.   From top to bottom, we make the following changes:

 

 

That's all.   We can run the query to see what effect it has.   The first image below shows the results of the 5x5 Unsharp computation.  The second image shows the original, zoomed to the same scale, for comparison.  Clearly, the 5x5 Unsharp result is sharper.

 

eg_process_images_3x3_19.png

5x5 Unsharp result, above.

 

eg_process_images_3x3_20.png

Original image, above.

 

Second Version: RGB 5x5 Unsharp Filter, Correcting RGB Artifacts

However, if we look closer we can see unexpected pixels of color in the Unsharp image result:

 

eg_process_images_3x3_21.png

 

The artifacts are caused by applying the same filter to three separated channels, with no guarantee that the results will fall within the range from 0 to 255 expected in an RGB image.  There are many sophisticated ways of normalizing filter results to fall within a 0 to 255 range, but the usual method is to simply trim the output to a range of 0 to 255.  Any values below 0 are set to 0, and any values above 255 are set to 255.  Primitive, but effective.

 

Given our modularlization of processing within functions, we can achieve that clipping by a simple re-write of the processtile function.   Instead of the original:

 

FUNCTION processtile(@t TILE, @r UINT8, @f TILE) TILE AS (

  TileFilter(@t, @r, @f)

) END;

 

We simply write:

 

FUNCTION processtile(@t TILE, @r UINT8, @f TILE) TILE AS (

  TileMin(TileMax(TileFilter(@t, @r, @f), 0), 255)

) END;

 

From our reading of the ever-helpful SQL Functions topic, we know that TileMax in the function above compares the output to 0 and takes the maximum.  Zero is always greater than any negative number, so any tile values lower than 0 are replaced by 0.   TileMin compares the output to 255 and takes the minimum.   255 is always less than anything greater than 255, so any tile values larger than 255 are replaced by 255.    Since we apply the same trimming to all the tiles we process, regardless of channel, we can do the trimming within the processtile function.

 

Making the above change, our final version of the RGB Unsharp query is:

 

-- $manifold$

--

 

VALUE @source_image TABLE = [st_peters];

VALUE @source_image_rect  NVARCHAR = '[ 0, 0, 1214, 862 ]';

 

-- Unsharp 5x5

VALUE @filter TILE = StringJsonTile('[

-0.00391, -0.01563, -0.02344, -0.01563, -0.00391,

-0.01563, -0.06250, -0.09375, -0.06250, -0.01563,

-0.02344, -0.09375,  1.85938, -0.09375, -0.02344,

-0.01563, -0.06250, -0.09375, -0.06250, -0.01563,

-0.00391, -0.01563, -0.02344, -0.01563, -0.00391

]', 5, 5, 1, true);

 

VALUE @radius UINT8 = 2;

 

CREATE TABLE [Custom] (

  [X] INT32,

  [Y] INT32,

  [Tile] TILE,

  [mfd_id] INT64,

  INDEX [mfd_id_x] BTREE ([mfd_id]),

  INDEX [X_Y_Tile_x] RTREE ([X], [Y], [Tile] TILESIZE (128,128) TILETYPE FLOAT64x3),

  PROPERTY 'FieldTileSize.Tile' '[ 128, 128 ]',

  PROPERTY 'FieldTileType.Tile' 'float64x3'

);

CREATE IMAGE [Custom Image] (

  PROPERTY 'Table' '[Custom]',

  PROPERTY 'FieldTile' 'Tile',

  PROPERTY 'FieldX' 'X',

  PROPERTY 'FieldY' 'Y',

  PROPERTY 'Rect' @source_image_rect

);

 

FUNCTION processtile(@t TILE, @r UINT8, @f TILE) TILE AS (

  TileMin(TileMax(TileFilter(@t, @r, @f), 0), 255)

) END;

 

FUNCTION processRGB(@t TILE, @r UINT8, @f TILE) TILE AS (

TileChannelsConcat(

  processtile(TileChannel(@t,0), @r, @f),

TileChannelsConcat(

  processtile(TileChannel(@t,1), @r, @f),

  processtile(TileChannel(@t,2), @r, @f)))

) END;

 

PRAGMA ('progress.percentnext' = '100');

INSERT INTO [Custom] (

  [X], [Y],

  [Tile]

) SELECT

  [X], [Y],

  CASTV ((TileRemoveBorder(processRGB(

    TileCutBorder(@source_image, VectorMakeX2([X], [Y]), @radius), @radius, @filter

    ), @radius)) AS FLOAT64)

FROM @source_image

THREADS SystemCpuCount();

ALTER TABLE [Custom] (

  ADD PROPERTY 'FieldCoordSystem.Tile' ComponentCoordSystem(@source_image)

);

TABLE CALL TileUpdatePyramids([Custom Image]);

 

Running the query, we see the result is a razor-sharp image with no color artifacts:

 

eg_process_images_3x3_22.png

 

Zooming into the dome, we see none of the unexpected colors seen before:

 

eg_process_images_3x3_23.png

 

An RGB Dual 3x3 Filter Prewitt Edge Detection Query

The example query now handles RGB images without generating color artifacts.   It uses a single 5x5 matrix filter.   We can easily adapt it to handle two 3x3 filters for a Prewitt edge detection filter.   We make changes very similar to those discussed in the SQL Example: Process Images using Dual 3x3 Filters topic:

 

-- $manifold$

--

 

VALUE @source_image TABLE = [st_peters];

VALUE @source_image_rect  NVARCHAR = '[ 0, 0, 1214, 862 ]';

 

-- Prewitt Edge Detect Filter 1

VALUE @filter1 TILE = StringJsonTile('[ -1, 0, 1, -1, 0, 1, -1, 0, 1 ]', 3, 3, 1, true);

 

-- Prewitt Edge Detect Filter 2

VALUE @filter2 TILE = StringJsonTile('[ -1, -1, -1, 0, 0, 0, 1, 1, 1 ]', 3, 3, 1, true);

 

VALUE @radius UINT8 = 1;

 

CREATE TABLE [Custom] (

  [X] INT32,

  [Y] INT32,

  [Tile] TILE,

  [mfd_id] INT64,

  INDEX [mfd_id_x] BTREE ([mfd_id]),

  INDEX [X_Y_Tile_x] RTREE ([X], [Y], [Tile] TILESIZE (128,128) TILETYPE FLOAT64x3),

  PROPERTY 'FieldTileSize.Tile' '[ 128, 128 ]',

  PROPERTY 'FieldTileType.Tile' 'float64x3'

);

CREATE IMAGE [Custom Image] (

  PROPERTY 'Table' '[Custom]',

  PROPERTY 'FieldTile' 'Tile',

  PROPERTY 'FieldX' 'X',

  PROPERTY 'FieldY' 'Y',

  PROPERTY 'Rect' @source_image_rect

);

 

FUNCTION processtile2f(@t TILE, @r UINT8, @f1 TILE, @f2 TILE) TILE AS (

  TileMin((TileMax(((TileFilter(@t, @r, @f1)^2 + TileFilter(@t, @r, @f2)^2)^0.5), 0)), 255)

) END;

 

FUNCTION processRGB2f(@t TILE, @r UINT8, @f1 TILE, @f2 TILE) TILE AS (

TileChannelsConcat(

  processtile2f(TileChannel(@t,0), @r, @f1, @f2),

TileChannelsConcat(

  processtile2f(TileChannel(@t,1), @r, @f1, @f2),

  processtile2f(TileChannel(@t,2), @r, @f1, @f2)))

) END;

 

PRAGMA ('progress.percentnext' = '100');

INSERT INTO [Custom] (

  [X], [Y],

  [Tile]

) SELECT

  [X], [Y],

  CASTV ((TileRemoveBorder(processRGB2f(

    TileCutBorder(@source_image, VectorMakeX2([X], [Y]), @radius), @radius, @filter1, @filter2

    ), @radius)) AS FLOAT64)

FROM @source_image

THREADS SystemCpuCount();

ALTER TABLE [Custom] (

  ADD PROPERTY 'FieldCoordSystem.Tile' ComponentCoordSystem(@source_image)

);

TABLE CALL TileUpdatePyramids([Custom Image]);

 

The changes we make to the query enable us to use two filters:

 

 

Having made the above changes we can run the query to see what happens:

 

eg_process_images_3x3_24.png

 

The tinted edge lines seen above are the usual, expected, otherworldly appearance of Prewitt edge detection filters applied to RGB images.  They are not pixel noise artifacts like those which appear if we do not trim the output tile channels to a range of 0 to 255

 

By changing the name of the image and the Rect values, we can apply the filter to different images.

 

eg_process_images_3x3_sample.png

Above is the original image.

 

eg_process_images_3x3_25.png

 

Above is the result of the dual-filter RGB Prewitt query, illustrating the remarkably clean edges extracted by the calculation.

 

eg_process_images_3x3_butterfly.png

Another example, using an RGB image of a butterfly, above.

 

eg_process_images_3x3_26.png

As processed by the RGB Prewitt filter.  Open a larger version of the above image.

 

Additional Examples

The following examples have the original image first, with the RGB Prewitt filtered image adjacent or below.

 

eg_process_images_3x3_b2.pngeg_process_images_3x3_27.png

A military intelligence application.

 

eg_process_images_3x3_houses.pngeg_process_images_3x3_28.png

Preparing an image for vectorization.

 

eg_process_images_3x3_interchange.png

eg_process_images_3x3_29.png

Object detection and roads vectorization.

 

eg_process_images_3x3_resort.png

eg_process_images_3x3_30.png

Creating vector drawings of facilities.

 

Videos

Manifold Viewer - How Matrix Filters Work - The easy, simple way to learn how filters work! Watch this action-packed video using Manifold Viewer that illustrates how matrix filters, also known as convolution filters, work. In addition to explaining filters, the video provides a real-life look at simple Manifold techniques for moving objects around in drawings using the Shift transform, and fast and easy use of Selection and tables to quickly put desired values into attributes. Sound technical? Sure, but in a very easy and simple way.

 

Manifold Viewer - Create Custom GPU Accelerated Filters in Seconds - A technical video using the free Viewer showing how to create your own, fully custom, fully GPU-parallel, convolution matrix filters, including Emboss, Sobel, Prewitt, and Kirsch edge detection and many more, for use in Viewer or Release 9. Modify the spatial SQL examples in the downloadable example project to specify a custom matrix and in seconds your custom filter can do image processing at GPU-parallel speeds. Viewer is read-only, but you can copy and paste the query text for custom filters to and from Notepad or any other text editor. Download the Custom_Filter_Examples.mxb sample project to try out the video in Viewer or Release 9.

 

Manifold Viewer - Speed Demo with 1280 GPU Cores - 2 Minutes vs 5 Days - Watch the free Manifold Viewer run CPU parallel and GPU parallel with 8 CPU cores and 1280 GPU cores to absolutely crush a job, doing in 2 minutes what would take non-GPU-parallel software 5 days. The video shows Viewer instantly pop open a 4 GB project that contains a huge, multi-gigabyte terrain elevation surface for Crater Lake, Oregon. With a point and click - no parallel code required - we compute the mean curvature at each pixel of the surface using a 7x7 matrix in under two minutes. We combine that with the original surface for enhanced hill shaded effects to better see details. Using an 11x11 matrix takes just over two minutes, a huge computation that takes days in non-GPU-parallel GIS packages.

 

See Also

Images

 

Tables

 

Data Types

 

Transform Pane

 

How Matrix Filters Work

 

Command Window

 

SQL Functions

 

SQL Example: Process Images with 3x3 Filters -  Shows a step-by-step example of developing an SQL query that takes a query written by the Edit Query button and then modifies that query into a general purpose query that can apply any 3x3 filter.   This makes it easy to use matrix filters we find on the web for custom image processing.   We extend the query by using parameters and adding a function, and then show how it can be adapted to use a 5x5 filter.

 

SQL Example: Process Images using Dual 3x3 Filters  - A continuation of the above topic, extending the example query to utilize two filters for processing, as commonly done with Sobel and Prewitt two filter processing.

 

SQL Example: Create NDVI Displays - How to create a query that creates an NDVI display from a four-band NAIP image, with tips and tricks on how to copy and paste existing information to get the result we want.

 

Example: Enhance Terrain with Curvatures -  We enhance a terrain showing Crater Lake, Oregon, by using mean curvature calculation to bring out details.   The example uses a 4 GB project containing a large terrain elevation surface.  Using a point-and-click dialog with no SQL, we apply automatic CPU parallelism and GPU parallelism to absolutely crush a task in two and a half minutes that would take non-parallel software days.

 

Example: Rearrange Channels using an Expression - We use a simple expression in the Transform pane to rearrange the order of channels within the data.