Backstory

TNRIS has a lot of historical imagery. It is the official archive for the State of Texas (and Texas is big). We're talking filiing cabinets, almost ceiling tall, filled with millions of physical aerial photos dating back almost a hundred years. They exist as a resource for anybody to come utilize. Although, naturally it is a little difficult to find what you're looking for (or at) as you sift through it all. Call me crazy but with today's technology I think the whole archive could be digital and freely available for the public to navigate, download, and view within a web map.

Hell, if companies like Planet are processing and serving global satellite imagery on a daily basis I think a few millions black-n-white photos from a dirty basement is... manageable.

Sure, it will be a lot of work but the hard part is getting it digital and georeferenced. If we can get a lambda raster processing machine up and running, we can make everything already digital immediately available. Then as every new photo frame gets scanned, it can be turned into a COG, cataloged, and served without any extensive work beyond the simple and conscious act of scanning (and georeferencing) the photo.

Processing Pipeline

The project github repo can be found here: https://github.com/TNRIS/lambda-s4

So with the backstory in mind, I conceptualized the entire process as a series of lambda functions chained together. The project repo consists of numerous directories named to include their numerical order and function within the process. Each directory is a step in the engine and serves a specific functionality (which the next step is dependent upon). Inside the base of the repo is 'exploration_instructional.md' which is a super messy documentation of every command I ran as I was working out the process as a whole. I left it in there simply for reference to the raw tests and manual steps used to prove the concept before cleaning it all up for documented deployment.

1. RDC (Research and Distribution Center) employees upload scanned image to appropriate `.../scanned/...` directory in the tree for storage. In reality, the 'upload' is a copy/paste to a FUSE s3fs directory mounted s3 bucket. No other event happens; this is just to make available the raw, scanned image in case that is a desired product for clients to download.
2. RDC employees upload georeferenced image to appropriate `.../georef/...` directory in the tree. In reality, the 'upload' is a copy/paste to a FUSE s3fs directory mounted s3 bucket. This triggers the preliminary reprojection lambda function by an event wired to monitor the bucket for all tif extensions.
3. The preliminary (technically first) lambda function `ls4-00-reproject` runs a verification of the GeoTiff's projection as uploads should already be in EPSG 3857 WGS84 Web Mercator Auxillery Sphere. But of course, people make mistakes so this premliminary check either starts the process by setting the correctly projected GeoTiff into a deliberate sub-directory or reprojects and then sets it aside. Once in the proper projection and sub-directory, this function invokes the first processing lambda.
• Uses rasterio with ManyLinux Wheels
• python 3.6 runtime
• Environment Variables:
  epsgSubDir='epsg3857/' (subdirectory for key upload; must be same as ls4-01-compress)
georefSubDir='deflate/' (subdirectory of uploaded input GeoTiff; must be same as ls4-01-compress and ls4-02-overviews)
4. The first lambda function `ls4-01-compress` runs generic DEFLATE compression on georeferenced tif and reuploads to same key but in a sub directory (environment variable defined). Then it invokes the second lambda directly (doesn't use event monitoring it's upload because you cannot duplicate trigger on mulitple lambdas and the output is also a .tif).
• Uses gdal_translate binary
• NodeJS 6.10 runtime
• Uploaded image key declares 'bw' or 'nc' (nc means natural color - or, RGB) so the function uses different band declarations when calling the gdal translate command
• Environment Variables:
  gdalArgs='-co tiled=yes -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co NUM_THREADS=ALL_CPUS -co COMPRESS=DEFLATE -co PREDICTOR=2' (gdal translate arguments)
  uploadBucket='project-bucket-name' (project bucket name)
  uploadKeyAcl='private' (output s3 upload acl)
  bwBands='-b 1' (band arguments for single band raster)
  ncBands='-b 1 -b 2 -b 3' (band arguments for nc raster)
  georefSubDir='deflate/' (subdirectory for key upload; must be same as ls4-00-reproject and ls4-02-overviews)
epsgSubDir='epsg3857/' (subdirectory for key upload; must be same as ls4-00-reproject)
5. The second lambda function `ls4-02-overviews` creates overviews on the compressed tif and dumps them alongside it in the sub directory (.ovr). This function has a sub directory environment variable which it verifies is part of the compressed tif key in order to run -- this means the sub directory environment variable for both functions must be the same. This triggers the third lambda function by an event wired to monitor the bucket for all ovr extensions.
• Uses gdaladdo binary
• NodeJS 6.10 runtime
• Environment Variables:
  uploadBucket='project-bucket-name' (project bucket name)
  gdaladdoLayers='2 4 8 16 32 64' (gdal addo layer arguments)
  gdaladdoArgs='-r average -ro' (gdal addo arguments)
  georefSubDir='deflate/' (subdirectory for key upload; must be same as ls4-00-reproject and ls4-01-compress)
6. The third lambda function `ls4-03-cog` creates the cloud optimized geotiff (COG) from tif and ovr in the sub directory. Then it invokes the fourth lambda directly (doesn't use event because you cannot duplicate trigger on mulitple lambdas and the output is also a .tif).
• Uses gdal_translate binary
• NodeJS 6.10 runtime
• Uploaded image key declares 'bw' or 'nc' (nc means natural color - or, RGB) so the function uses different byte and compression tyeps when calling the gdal translate command
• Environment Variables:
  uploadBucket='project-bucket-name' (project bucket name)
  uploadKeyAcl='public-read' (output s3 upload acl)
  bwGdalArgs='-of GTiff -ot Byte -a_nodata 256 -co TILED=YES -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co COMPRESS=DEFLATE -co COPY_SRC_OVERVIEWS=YES --config GDAL_TIFF_OVR_BLOCKSIZE 512' (gdal translate arguments for single band raster)
  ncGdalArgs='-of GTiff -co TILED=YES -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co COMPRESS=JPEG -co JPEG_QUALITY=85 -co PHOTOMETRIC=YCBCR -co COPY_SRC_OVERVIEWS=YES --config GDAL_TIFF_OVR_BLOCKSIZE 512' (gdal translate arguments for nc raster)
  georefSubDir='deflate/' (subdirectory for key upload; must be same as ls4-02-overviews)
7. The fourth lambda function `ls4-04-shp_index` creates the shapefile tile index of all COGs in s3 for the collection and drops it off in s3. Then it uploads a copy of the tile index to a new table in a PostGIS RDS for the Mapserver mapfile to use. This function is special insofar as accessing the RDS; the RDS is within a VPC so the lambda function must be within the same VPC to access it (security!). When a lambda function is deployed within a VPC, it no longer has access to S3 except through a VPC Endpoint. So, a VPC endpoint is deployed alongside this function with the function, RDS, and endpoint all residing in (or pointing to) the same subnets. This function triggers the fifth lambda function by an event wired to monitor the bucket for all .shp extensions.
• Uses rasterio with ManyLinux Wheels
• python 3.6 runtime
• Environment Variables:
  DB_DRIVER='postgresql' (sqlalchemy database driver)
  DB_NAME='database-name' (database name)
  DB_USER='username' (user with table create, drop, and update permissions)
  DB_PASSWORD='password' (user password)
  DB_HOST='host connection url' (RDS host url)
  DB_PORT='5432' (database port. postgres default is 5432)
8. The fifth lambda function `ls4-05-mapfile` creates the mapfile for the collection and drops it off in s3. This is accomplished by using a template ('template.map') mapfile previously setup for WMS with placeholders for python to overwrite with the specifics related to the collection. Placeholders are variable names surrounded by less-than and greater-than carrots. Psycopg2 is used to query the collection's tile index in PostGIS for the 'EXTENT' x/y minimums and maximums. The mapfile does require an AWS user access key and secret access key. These are used to access s3 to retrieve the COGs. I setup a new user with only s3 permissions to the project for use in these mapfiles and by the Mapserver EC2 for mounting the bucket with FUSE s3fs (see below). Note: the mapfiles are uploaded with specific headers declaring the owner and permissions of the file so it can be read by Mapserver (see below).
• Uses standard python packages; no special binaries
• python 3.6 runtime
• Environment Variables:
  DB_NAME='database-name' (database name)
  DB_USER='username' (user with table create, drop, and update permissions)
  DB_PASSWORD='password' (user password)
  DB_HOST='host connection url' (RDS host url)
  DB_PORT='5432' (database port. postgres default is 5432)
  MAPSERVER_ACCESS_KEY_ID='' (mapserver user's access key id)
  MAPSERVER_SECRET_ACCESS_KEY='' (mapserver user's seecret access key)
DNS_URL='http://server.yourdomain.com' (mapserver instance dns url)

Mapserver

Okee dokee, so the processing pipeline is setup and successfully converts individually uploaded image frames into COGS while continually regenerating (in order to update with new frames) the tile index of all frames and a mapfile to serve them out as a WMS Service. The other half of the project is the Mapserver to actually host the services by reading the mapfiles and serving out the COGs from s3. The step-by-step details of setting up such a Mapserver are outlined in my post FUSE s3 Mapserver but here I'll just provide an overview of the main points to know and consider.

• Started with an Amazon OS ECS Optimized AMI with Docker since I would be running Mapserver within a container. Originally spun up a basic micro EC2 for testing but once all the kinks were worked out, I re-provisioned a new AMI for use with ECS and spun up a new cluster to utilize it.
• Used these basic instructions to install fuse and mount s3 as a drive on the machine.
• FUSE s3fs uses an IAM User Key ID and Secret Key for permissions to connect to the s3 bucket so I created a user to represent the Mapserver. I created a custom permission policy to assign to this user with only permissions to read and write to only the project bucket. This user's key and secret can also be used in mapfiles for AWS bucket access to COGs.
• FUSE accesses the user key/secret with a .passwd-s3fs file located in the 'ec2-user' home directory. When setting up this file, be sure to chown the file to the 'ec2-user' user. Permissions instructions here.
• You'll have to sudo edit /etc/fuse.conf to uncomment out the 'user_allow_other' line to permit machine users to access the mounted s3 directory.
• You'll want to setup the directory to automatically mount on machine bootup. Probably good practice if managing a single EC2 but definitely a requirement if setting up an AMI for ECS since ECS machines may come and go. New ones will need the directory already mounted as they turn on.
• s3 is an object store but FUSE s3fs mounts as a directory and recognizes 'folders'. Therefore, the 'directory' where the mapfiles reside in the s3 bucket must be owned by the os user ('ec2-user') running the docker container. This is accomplished by mounting the bucket, then doing a simple mkdir to create the folder rather than creating it in the AWS Console (which creates them as 'root').
• Same as the mapfile directory owner, the actual '.map' mapfiles need to be owned by 'ec2-user' and have the proper permissions. Since we are creating our mapfiles programatically, this is accomplished by using the proper headers when uploaded to s3 within function 5 `ls4-05-mapfile`. Boto3 is used to accomplish this: import boto3 s3 = boto3.resource('s3') bucket_name = '<-- bucket name -->' upload_file = '/<-- path to file -->/<-- filename -->.map' upload_key = '<-- upload key with filename including .map -->' # example: 'testt/test2.map' s3.Bucket(bucket_name).upload_file(upload_file,upload_key,ExtraArgs={'Metadata':{'mode':'33204','uid':'500','gid':'500','mtime':'1528814551'}}) It is within the ExtraArgs - Metadata that we can apply the required 'mode', 'uid', 'gid', and 'mtime' of the uploaded mapfile. The user and group IDs (uid, gid) should be that of the OS user 'ec2-user'. It seems standard and reliable to me that Amazon OS default user 'ec2-user' is UID and GID 500.
• Spin up mapserver docker with a -v volume flag passing the machine FUSE mounted directory to the docker. Example: sudo docker run --detach -v /home/ec2-user/<-- mounted bucket -->/<-- bucket folder -->:/mapfiles:ro --publish 8080:80 --name mapserver camptocamp/mapserver:7.4 After the docker is running, setup an error log file with these commands:
  1. sudo docker exec mapserver touch /var/log/ms_error.log
  2. sudo docker exec mapserver chown www-data /var/log/ms_error.log
  3. sudo docker exec mapserver chmod 644 /var/log/ms_error.log
  4. Then you can use sudo docker exec mapserver cat /var/log/ms_error.log to view the logs
• Alternative to the previous point, create a custom docker image to utilize (opposed to DockerHub's "camptocamp/mapserver:7.4") which already has the error log file already provisioned. This is required if deploying to ECS.
• Template WMS Service Mapfile

AWS Architecture

• s3 bucket which holds the rasters/COGs, tile index shapfiles, and mapfiles.
• Mapserver docker instance (some details above but super details here). Initially for discovery and testing, the docker was spun up on a plain AWS OS EC2 and when it was completely configured it was the basis for the custom AMI to be used by ECS. In production, use a MapServer docker running in ECS on an EC2 with custom AMI that has a FUSE s3fs pointing a directory at the COG/Mapfile s3 bucket. This includes:
  • ECS Cluster running necessary machine (we use r3 large) with custom AMI
  • ECS Service in the cluster with a task definiton containing the proper port mappings from the machine to the docker, proper mounted /mapfile volume, and custom docker image
  • Elastic Load Balancer pointing http traffic to the cluster machines
  • DNS record pointing toward the Elastic Load Balancer
  • ECR (container registry) to store custom docker image if using ECS
• IAM User with policy permissions to project s3 bucket for MapServer to use. Generate an Access Key ID and Secret Access Key for use as environment variables in function `ls4-05-mapfile`.
• Lambda IAM Role with full Lambda permissions and full s3 permissions to the project s3 bucket. Apply this role to every lambda function being mindful that function 4 is inside the VPC and needs additional subnet & VPC endpoint configuration.
• VPC Endpoint for Function 4 for s3 access since it is deployed within VPC to access RDS.
• Lambda function for each step in the process with appropriate environment variables, role, and event triggers as outlined above.
• Postgres RDS provisioned with PostGIS installed and a user for the lambda functions to utilize. The user should be granted permissions to create and alter tables and maintain ownership of the spatial_ref_sys PostGIS table.
• Appropriate networking and security amongst these cloud services will be required; the details of which I will not go through here, sorry.

Hurdles

The main hurdles that needed ironing out were:

• s3 key structures which organize the tifs so that scanners/georeferencers can drop off new images and the process can consistently handle them.
• setting up FUSE s3fs such that scanners/georeferencers can just drop off new images in a folder to let the process do it's thing. setting this up also lets the host mapserver use s3 for all the mapfiles. by hosting mapfiles from s3, the process can create new mapfiles and put them in a specific key structure which the mapserver automatically reads without redeployment of any kind. in short: new image uploaded = autmatically created new WMS. I detailed the process of creating this Mapserver in my post FUSE s3 Mapserver.
• almost the entire process uses GDAL binaries. this is a major issue when it comes to serverless lambda as the size of these binaries are waaaaaay too large for the compressed 50 MB function upload limit (or 250 MB uncompressed via the more forgiving s3 route).
  The first part of the process used indepedent GDAL Translate and GDAL Addo binaries, precompiled and supplied by Mark Korver (shoutout below) which allow the tif to COG conversion to be possible. These binaries are available and can be snatched from the function directory folder inside it's '/bin' subfolder.
  The second part of the process uses Rasterio with ManyLinux Wheels to do a python version of gdaltindex to create the tile index (shoutout below). Luckily, a rasterio python package has been in development to incorporate them - this was a life saving necessity. The package was still too large to deploy though until I read Seth Fitzsimmons' (shoutout below) clever hack to remove all unused python dependency files and shrink the deployment. I expanded on Seth's instruction with details and specific commands in my post Got Shrinkage?.
• When uploading multiple geotiffs simultaneously or in succession, the lambda functions trip up as rasterio (or gdal) tries to .open() them. It gives a headache worthy 'not a supported format' error. This is because GDAL's /viscurl/ and related systems (/vsis3/) cache directory listings and the new uploads are altering what is in the s3 'directory'. This is fixed by using the osGEO GDAL python package with ManyLinux Wheels and calling gdal.VSICurlClearCache() before running the file open command and/or after each s3 upload. More Info. It is because of this issue (discovered deeper into the testing of this engine) that I switched from Rasterio with ManyLinux Wheels to GDAL with ManyLinux Wheels. Rasterio has no method to clear the cache I could find.

Details related to the s3 key structure (repo wiki), setting up FUSE s3fs, Rasterio with ManyLinux Wheels, and shrinking of the lambda function for deployment can be found in the github repo README.

Shout Outs

• This process is extensively credited to Mark Korver of AWS and his initial series of lambda functions to process tifs and create COGs. He has presented this process several times including FOSS4G NA and the Texas GeoRodeo. His initial workshop to outline the process, with links to the sample functions (used as the base of this project), can be found here.
• A Rasterio python package with ManyLinux Wheels was an absolute necessity! Thanks to everyone making that a reality!
• Also due much credit, is Seth Fitzsimmons who is doing a ton of innovation with raster processing, hosting tiles from s3, and using lambda to do it. His code has been an excellent source of inspiration. His blog post on Slimming Down Lambda Deployment Zips was a real ground breaker for keeping this whole project serverless.