Skip to content

Jump to hands-on lesson

Image title

Background

The TIFF file format (Tagged Image File Format) is an old format dating back to 1992. TIFF are great for high-resolution verbatim raster images. TIFF are still used a bit in high-end photography, but where it has really grown a second life is in digital cartography. The variation called GeoTIFF has been widely adopted as a way to share satellite images and other satellite data.

While the GeoTIFF file format has long been thought of as only suitable for raw data: if you wanted to display it on a map, you’d convert it into tiles. If you wanted a static image, you’d render it into a PNG or JPEG. But Cloud-Optimized GeoTIFF means that GeoTIFFs can be a bit more accessible than they used to be.

Cloud Optimized GeoTIFF (COG)

A Cloud Optimized GeoTIFF (COG) is a regular GeoTIFF file, aimed at being hosted on a HTTP file server, with an internal organization that enables more efficient workflows on the cloud. In other words, we can stream the content of a GeoTIFF file from a server into a client application without having to download the full file.

COGs have three major features that are baked into the file:

Tiling

Tiles are small, regular, and independent parts of a larger map. They are typically 256x256 pixels in size. COGs have tiles explicitly specified in their format structure, whereas regular GeoTIFFs may not. Tiling is a way to speed up map display because only the tiles that are visible in the current view need to be loaded.

Overviews

Overviews are downsampled thumbnail images of the tile. A COG will have many overviews matched to each Zoom Level.

geotiff_pyramid

GeoTIFF pyramid by Zoom Level

HTTP(s) GET Range Request

The HTTP GET Range Request, also known as Byte Serving, allows a client to request specific chunks of the COG using a combination of the tiles and overviews. If you are zoomed into a specific portion of the COG, then you only request the tiles that are visible in the current view. This is the same technology that enables streaming of other media types like video and audio.

Check out the COG Specification

Example COGs on the internet

There are numerous cloud based data stores hosting COGs, take a look through a few of these:

Applications that can use COGs

COGs are geotiffs, so any software application that can read and work with geotiffs will be able to read and work with COGs. This includes QGIS, ArcGIS, and Google Earth Engine.

Creating Your Own COGs

Creating COGs can be accomplished using:

  • GDAL command line tool. See Hands On Lesson

  • Cogger is a rapid COG generator from GeoTIFF

  • In the python ecosystem rio-cogeo is a RasterIO plugin to create and validate COGs

If I have COGs, do I still need a tile server?

The breakthrough of COGs is that they have internal tiling which allows them to be streamed into applications without the need for an additional tile server. For most individuals and small organizations, this should be all you need.

However, there are a few reasons why you might still want to use a tile server with COGs:

  • Performance: Tile servers can cache tiles in memory, which can improve performance by reducing the number of times that tiles need to be read from disk.

  • Scalability: Tile servers can be scaled to handle large numbers of requests.

  • Security: Tile servers can be used to encrypt tiles, which can help to protect them from unauthorized access. If you are streaming COGs to a large number of clients or if you need to ensure that your tiles are secure, then I recommend using a tile server. However, if you are only streaming COGs to a small number of clients and you do not need to worry about security, then you can use the built-in tiles in COGs.

How are COGs different from XYZ and WMTS tiles?

Web mapping tile services (WMTS) and XYZ tiles are primarily designed for efficient map display in web environments. Their main goal is to provide quick and seamless map visualizations over the internet by serving small, pre-defined tiles at multiple zoom levels. These tiles are ideal for web maps where users might pan and zoom around the globe, as the small tiles can be fetched and displayed rapidly.

However, for analysis purposes – where users might want to compute statistics, apply algorithms, or extract detailed information from imagery or raster data – these tiling methods are not optimal. The reason is that analysis often requires access to raw, high-resolution data rather than the downsampled or potentially lossy representations provided by these tiles.

That's where formats like Cloud Optimized GeoTIFFs (COGs) come into play. COGs are designed to allow for efficient access to high-resolution raster datasets, making them more suited for analytical purposes. With COGs, one can access and process only specific portions of a large raster without downloading the entire file, making it efficient for cloud-based analysis workflows.

In summary, COGs are designed for efficient access to high-resolution raster data, while XYZ and WMTS tiles are designed for efficient map display.


Hands On

Stream COGs into QGIS

If you are unfamiliar with QGIS, here are 3 ways to use the open-source program

Install on your local machine

Download and install QGIS on your local machine

Use docker to run QGIS locally without installation

Educational material on software containers and Docker are here.

Installing Docker on your personal computer

We are going to be using virtual machines on the cloud for this course, and we will explain why this is a good thing, but there may be a time when you want to run Docker on your own computer.

Installing Docker takes a little time but it is reasonably straight forward and it is a one-time setup.

Installation instructions from Docker Official Docs for common OS and chip architectures:

Once docker has been installed on your local machine it is easiest to use it on the Command Line Interace (CLI). Run the following commands in your terminal:

this will allow the container access to the X server for display purposes

xhost +

this will run the QGIS container and open the application

docker run --rm -it --name qgis \
    -v $(pwd):/data \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e DISPLAY=unix$DISPLAY \
    qgis/qgis:release-3_34 qgis
Run cloud instance of QGIS on Cyverse

Cyverse is a cloud computing and storage tool housed at the University of Arizona. You can sign up for a user account here.

After getting the account, you need to request access to the Visual Interactive Computing Environment (VICE) app. This is a cloud-based desktop environment that allows you to run applications like QGIS in the cloud.

To request VICE access, visit the User Portal look for

and select the REQUEST ACCESS link.


Instructions for launching QGIS and other interactive apps within Cyverse are found here.

Now that you have QGIS running, let's stream a COG into the application.

  1. Open QGIS

  2. In the "Layers" then "Add Layer" and then "Add Raster Layer"

  3. Choose the Source Type and select "Protocol: HTTP(s), cloud, etc" for a file on your computer

  4. Enter a valid https:// in the URl field for a COG you found online
    Here are two COG examples to try. They are both drone orthomosaics that are located in the Cyverse Data Store:

https://data.cyverse.org/dav-anon/iplant/home/jgillan/USGA/imagery_products/hole_17_ortho_cog.tif

https://data.cyverse.org/dav-anon/iplant/home/jgillan/STAC_drone/22_2_ortho_cog.tif

Click "Add" and the COG will be streamed into QGIS qgis

An orthomosaic of a golf course should appear in QGIS

qgis

Create COGs with GDAL

Geospatial Data Abstration Library (GDAL) is an open source library for working with geospatial data. Most GIS software use GDAL. For regular users, GDAL is a command line tool.

The lastest versions of GDAL (>v3.1) have COG generator installed by default.

Install GDAL

GDAL installation can at times be difficult. When different older python environments are installed on a desktop or laptop GDAL can become broken or incompatiblity issues can come up when installing it.

USGS Windows GDAL Installation Guide

Official GDAL Install Guide

QGIS installs GDAL by default

Anaconda and its package management conda

Docker osgeo/gdal images are maintained on the Docker Hub

1 Download the example geotif file from the Cyverse Data Store. The file is a 4-band multispectral satellite image from Planet. It was captured on Sept. 6, 2020 over the city of Tucson, AZ and the Santa Catalina Mountains. The file is 1.59 GB in size.

2 Bring it into QGIS to view it.

planet

3 Use gdal to read details of the file

If you managed to install gdal on your local machine, you can run the following command in your terminal:

gdalinfo planet_6sept2020.tif


If you are using docker, you can run the following command in your terminal:

docker run --rm -v /Users:/Users ghcr.io/osgeo/gdal:alpine-small-3.8.0 gdalinfo $PWD/planet_6sept2020.tif


4 Use gdal to create a COG from the sample file

We will be using the gdal_translate command to create COGS. The gdal_translate command is used to convert raster data between different formats. Here is additional documentation on COG creation options (-co).


for local install:

gdal_translate -of COG -co COMPRESS=LZW planet_6sept2020.tif planet_6sept2020_COG.tif


If using docker:

docker run --rm -v /Users:/Users ghcr.io/osgeo/gdal:alpine-small-3.8.0 gdal_translate -of COG -co COMPRESS=LZW $PWD/planet_6sept2020.tif $PWD/planet_6sept2020_COG.tif


5 Inspect the COG

The new file (2.19GB) should be larger than the original file (1.59GB). This is because the COG has internal tiling and overviews.

Let's look at the details of the new file

gdalinfo planet_6sept2020_COG.tif

or

docker run --rm -v /Users:/Users ghcr.io/osgeo/gdal:alpine-small-3.8.0 gdalinfo $PWD/planet_6sept2020_COG.tif

Buried in all the metadata are a few lines that tell us that the file is a COG. The layout is stated as 'COG' and 'overviews' are listed for each band. 

 Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=PIXEL
  LAYOUT=COG

 Band 1 Block=512x512 Type=UInt16, ColorInterp=Gray
  Min=0.000 Max=8006.000 
  Minimum=0.000, Maximum=8006.000, Mean=712.188, StdDev=549.828
  NoData Value=65536
  Overviews: 9414x7918, 4707x3959, 2353x1979, 1176x989, 588x494, 294x247

Use Cyverse to Store and Share Your COG

Graphic User Interface (GUI)

  • Create a new folder

  • Go into the new folder
  • Drag and drop the COG into the folder
  • Navigate one folder up
  • Click the 3dot menu to the far right of your new folder and select "Share"

  • Type 'anonymous' in the search bar and select the 'anonymous' user. This will make your COG publically available and able to stream out.

  • Once you COG has been uploaded, please click on the 3dot menu to the far right of your COG and select "Public Link"

  • Copy the link and paste it into the CogEO Viewer to view your COG

  • Alternatively, you can paste the link into QGIS to stream the COG into the application

Additional Resources

COGS in Production blog post by Sean Rennie

Blog on COGs

Mapscaping on COGS

Intro to COGs from Planet

Last update: 2023-12-05