Aws batch hillshade - #2
Open
MDavidson17 wants to merge 6 commits into
Open
Conversation
and increase job size to avoid too many requests exception
blacha
reviewed
Aug 31, 2021
| RUN rm /bin/sh && ln -s /bin/bash /bin/sh | ||
|
|
||
| # install awscli | ||
| RUN apt update && apt install awscli -y |
Member
There was a problem hiding this comment.
combine apt installs into one install apt install awscli jq -y
its also a good idea to clean up the packages after install, I cant remember the exact command but its something like && apt-cache-clean
blacha
reviewed
Aug 31, 2021
| @@ -0,0 +1,32 @@ | |||
| #!/bin/bash | |||
| cd ./app | |||
| tmp_dir=$(mktemp -d -t hs-XXXXXXXXXX) | |||
Member
There was a problem hiding this comment.
generally use long form arguments to make it easier to read -d vs --directory
blacha
reviewed
Aug 31, 2021
| @@ -0,0 +1,63 @@ | |||
| import cdk = require('@aws-cdk/core'); | |||
Member
There was a problem hiding this comment.
never use require
import * as cdk from '@aws-cdk/core
blacha
reviewed
Aug 31, 2021
| let string = '' | ||
| for await (let file_name of fsa.list(ReadFromFolder)) { | ||
| // let re = /\ /gi; | ||
| // file_name = file_name.replace(re, '\\ ') |
Member
There was a problem hiding this comment.
would be a good idea to simplify this down
const goodExtension = new Set('tif', 'tiff', 'asc')
cosnt ext = extname(file_name)
if (ext in file_extensions) {
file_name.replace(ext, 'tiff')
}
blacha
reviewed
Aug 31, 2021
| fileList = fileList.concat(string) | ||
| count++ | ||
| } | ||
| if (count == 25) { |
Member
There was a problem hiding this comment.
const RecordsPerJob = 25
if (fileList.length >= RecordsPerJob)
blacha
reviewed
Aug 31, 2021
| } | ||
| } | ||
| if (count > 0) { | ||
| submit(correlationId, "final-upload-" + upload_count, fileList) |
blacha
reviewed
Sep 1, 2021
| let base = '' | ||
| let string = '' | ||
| for await (let file_name of fsa.list(ReadFromFolder)) { | ||
| // let re = /\ /gi; |
Member
There was a problem hiding this comment.
The escaping is a problem in bash, I would leave these names alone
added await submit fix handling of file list convert to python script tidy up and delete temp_dir added logging
replaced by python script
paulfouquet
suggested changes
Nov 1, 2021
paulfouquet
left a comment
paulfouquet
left a comment
Contributor
There was a problem hiding this comment.
Does the script work with the bad indentation?
| os.environ["AWS_SECRET_ACCESS_KEY"]=credentials['SecretAccessKey'] | ||
| os.environ["AWS_SESSION_TOKEN"]=credentials['SessionToken'] | ||
|
|
||
| temp_dir = tempfile.mkdtemp() |
Contributor
There was a problem hiding this comment.
The indentation is weird here?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
batch process to create hillshades.
not included: