DEV Community: Breadcentric

How to add a service to AWS CLI and boto3

Breadcentric — Fri, 01 Jan 2021 21:41:38 +0000

In this post I will share how to add an AWS service to AWS CLI utility and for use with boto3.

One of the missing features AWS DeepRacer Community have been regularly raising has been lack of CLI access to AWS DeepRacer Console. Thanks to Don Barber I have finally understood how AWS CLI and Boto3 library work and would like to share it with you.

Icon image used shared under Creative Commons here

What is AWS CLI

AWS Command Line Interface is a utility that you can use to interact with AWS services through a terminal. It supports many of the services that Amazon Web Services Provide:

accessanalyzer                           | acm                                     
acm-pca                                  | alexaforbusiness                        
amp                                      | amplify                                 
amplifybackend                           | apigateway                              
apigatewaymanagementapi                  | apigatewayv2                            
appconfig                                | appflow                                 
appintegrations                          | application-autoscaling                 
application-insights                     | appmesh                                 
appstream                                | appsync                                 
athena                                   | auditmanager                            
autoscaling                              | autoscaling-plans                       
backup                                   | batch                                   
braket                                   | budgets                                 
ce                                       | chime                                   
cloud9                                   | clouddirectory                          
cloudformation                           | cloudfront                              
cloudhsm                                 | cloudhsmv2                              
cloudsearch                              | cloudsearchdomain                       
cloudtrail                               | cloudwatch                              
codeartifact                             | codebuild                               
codecommit                               | codeguru-reviewer                       
codeguruprofiler                         | codepipeline                            
codestar                                 | codestar-connections                    
codestar-notifications                   | cognito-identity                        
cognito-idp                              | cognito-sync                            
comprehend                               | comprehendmedical                       
compute-optimizer                        | connect                                 
connect-contact-lens                     | connectparticipant                      
cur                                      | customer-profiles                       
databrew                                 | dataexchange                            
datapipeline                             | datasync                                
dax                                      | deepracer                               
detective                                | devicefarm                              
devops-guru                              | directconnect                           
discovery                                | dlm                                     
dms                                      | docdb                                   
ds                                       | dynamodb                                
dynamodbstreams                          | ebs                                     
ec2                                      | ec2-instance-connect                    
ecr                                      | ecr-public                              
ecs                                      | efs                                     
eks                                      | elastic-inference                       
elasticache                              | elasticbeanstalk                        
elastictranscoder                        | elb                                     
elbv2                                    | emr                                     
emr-containers                           | es                                      
events                                   | firehose                                
fms                                      | forecast                                
forecastquery                            | frauddetector                           
fsx                                      | gamelift                                
glacier                                  | globalaccelerator                       
glue                                     | greengrass                              
greengrassv2                             | groundstation                           
guardduty                                | health                                  
healthlake                               | honeycode                               
iam                                      | identitystore                           
imagebuilder                             | importexport                            
inspector                                | iot                                     
iot-data                                 | iot-jobs-data                           
iot1click-devices                        | iot1click-projects                      
iotanalytics                             | iotdeviceadvisor                        
iotevents                                | iotevents-data                          
iotfleethub                              | iotsecuretunneling                      
iotsitewise                              | iotthingsgraph                          
iotwireless                              | ivs                                     
kafka                                    | kendra                                  
kinesis                                  | kinesis-video-archived-media            
kinesis-video-media                      | kinesis-video-signaling                 
kinesisanalytics                         | kinesisanalyticsv2                      
kinesisvideo                             | kms                                     
lakeformation                            | lambda                                  
lex-models                               | lex-runtime                             
license-manager                          | lightsail                               
location                                 | logs                                    
lookoutvision                            | machinelearning                         
macie                                    | macie2                                  
managedblockchain                        | marketplace-catalog                     
marketplace-entitlement                  | marketplacecommerceanalytics            
mediaconnect                             | mediaconvert                            
medialive                                | mediapackage                            
mediapackage-vod                         | mediastore                              
mediastore-data                          | mediatailor                             
meteringmarketplace                      | mgh                                     
migrationhub-config                      | mobile                                  
mq                                       | mturk                                   
mwaa                                     | neptune                                 
network-firewall                         | networkmanager                          
opsworks                                 | opsworkscm                              
organizations                            | outposts                                
personalize                              | personalize-events                      
personalize-runtime                      | pi                                      
pinpoint                                 | pinpoint-email                          
pinpoint-sms-voice                       | polly                                   
pricing                                  | qldb                                    
qldb-session                             | quicksight                              
ram                                      | rds                                     
rds-data                                 | redshift                                
redshift-data                            | rekognition                             
resource-groups                          | resourcegroupstaggingapi                
robomaker                                | route53                                 
route53domains                           | route53resolver                         
s3control                                | s3outposts                              
sagemaker                                | sagemaker-a2i-runtime                   
sagemaker-edge                           | sagemaker-featurestore-runtime          
sagemaker-runtime                        | savingsplans                            
schemas                                  | sdb                                     
secretsmanager                           | securityhub                             
serverlessrepo                           | service-quotas                          
servicecatalog                           | servicecatalog-appregistry              
servicediscovery                         | ses                                     
sesv2                                    | shield                                  
signer                                   | sms                                     
sms-voice                                | snowball                                
sns                                      | sqs                                     
ssm                                      | sso                                     
sso-admin                                | sso-oidc                                
stepfunctions                            | storagegateway                          
sts                                      | support                                 
swf                                      | synthetics                              
textract                                 | timestream-query                        
timestream-write                         | transcribe                              
transfer                                 | translate                               
waf                                      | waf-regional                            
wafv2                                    | wellarchitected                         
workdocs                                 | worklink                                
workmail                                 | workmailmessageflow                     
workspaces                               | xray                                    
s3api                                    | s3                                      
configure                                | deploy                                  
configservice                            | opsworks-cm                             
runtime.sagemaker                        | history

The tool uses Amazon's boto3 library which allows for a programatic access to those services in Python. Boto3 in turn uses botocore as a dependency that contains definitions of the services.

How are service endpoints defined in AWS CLI?

Each service has at least a Service Definition File which is written in JSON and contains a few pieces of information:

version - version of the service definition file
metadata - additional information about interacting with the service
operations - what requests can be sent to the service
shapes - operations inputs and outputs
documentation - pieces of data that are used to form documentation in aws <command> help

This is not the only file available and you can read about more of them in loaders.py docstring. They are not needed in my case.

The SDF file needs to be present in one of the following locations top be used by AWS CLI or boto3:

<botocore root>/data/
~/.aws/models
A path added to a list in the AWS_DATA_PATH env var

If you're using boto3, you can also use code to add your location to the default loader.

Can I create my own service definition?

I think not. I was actually planning to do that myself but then I have realised that you need to know the exact service API endpoint and version to be able to work with it. Once you do you can observe what your service console page is calling on the api gateway and do the same yourself.

But I guess that's not the use case AWS are after. When the service authors decide their API is ready they can expose it together with the version, they can provide the service definition files for people with access to preview and they can then use the API. If you're in contact with the dev team you can try some pretty-please approach and maybe they will share it with you.

AWS DeepRacer CLI

If you haven't yet seen Don's reInvent 2020 talk "Shift your ML model into overdrive with AWS DeepRacer analysis tools" I recommend that you do. As part of preparation for the talk Don has prepared some updates to log analysis tool provided by AWS. The tool has been there from day one together with DeepRacer workshops. I've looked at this new line:

region = "us-east-1"
dr_client = boto3.client('deepracer', region_name=region,
        endpoint_url="https://deepracer-prod.{}.amazonaws.com".format(region))
models = dr_client.list_models(ModelType="REINFORCEMENT_LEARNING",MaxResults=100)["Models"]

And thought: "cool, we can now use aws cli!". Lesson learned: don't skip half of the code provided when trying to get something to work. Right above this code Done does this:

envroot = os.getcwd()
aws_data_path = set(os.environ.get('AWS_DATA_PATH', '').split(os.pathsep))
aws_data_path.add(os.path.join(envroot, 'models'))
os.environ.update({'AWS_DATA_PATH': os.pathsep.join(aws_data_path)})

And there it was: The service definition file for DeepRacer.
I am pleased to say that I have added the service definition file to deepracer-utils 0.8. If you would like to use it with boto3, simply run:

from deepracer.boto3_enhancer import add_deepracer

client = deepracer_client()

dir(client)

I'm leaving it out there since you can already start playing with it, but expect updates to install it for use in AWS CLI and for some handy use cases, examples and perhaps even docs :)

Teach me, Amazon CodeGuru

Breadcentric — Mon, 07 Dec 2020 10:24:55 +0000

Today I will run a machine-learning-backed code analysis against the Python-based deepracer-utils project.

As the AWS re:Invent continues, many new products have been announced. I was especially impressed with Andy Jassy's Keynote on Tuesday Jan 1st, it was pretty fast pace and packed with announcements. Blink once, you'll miss two new products. Interestingly machine learning was largely omitted since there is a separate Machine Learning Keynote with Swami Sivasubramanian on Dec 8th at 2:45 pm UTC.

I'm especially curious about the devices announcements. DeepRacer is still holding up pretty damn well with this year's final in progress (virtually), multiple companies using it for mass machine learning and cloud immersion (DBS - 3K+ employees, Accenture - 2K+, DNP - 1K+, and there's more that I cannot mention :) ) and while last years AWS DeepComposer did not make many jaws drop it still was pretty fun for learning. I'm wondering if there will be any additions to this group.

AWS CodeGuru Reviewer

CodeGuru Reviewer is a code analysis utility provided by AWS. It initially provided analysis for Java, this year Python was added. I haven't used it before, mainly because all of my Java code is work code and I haven't had much to run it against. This year however I have published deepracer-utils to support various methods of training that little rascal.

I will attach it to a GitHub repository and see what analysis results I get.

Deepracer-utils

Deepracer-utils is a Python based project, a first one in which I published on PyPi and prepared with my most complete (at that time) understanding of what a Python project should look like.

My expectations

I am expecting to get some proper hints about what I have flipped up, hopefully no critical level issues. I have 10 years professional experience but since Java ways don't always translate well into Python, I'm hoping to learn a thing or two today.

Getting started

The main project page has a quick access to a setup page:

Then you need to grant access to your repositories. The tool is asking for an admin access to webhooks (to integrate to selected projects) and gives you the option to do it for selected organisations (you need to click the "Grant" button next to the org for permission to be granted). Another permission is read access to both public and private repositories.

If you would like to revoke the authorization, you will find it in your Applications setting page on the Authorized OAuth Apps.

I have granted the permissions, but aws-deepracer-community projects where nowhere to be found - I decided to revoke the authorization and redo it since I could not edit the nature of an already granted authorization (guess it makes sense). That did work. I selected the right repo and presto:

In the repository itself I can see a new webhook created, listening for Pull requests, Pull requests reviews and Pull request review comments.

The associated repositories section doesn't give you many options: you can associate repository, disassociate repository or manage repository's tags. I go over to Code review page and there are two tabs, one for pull requests (empty since I made no PRs today) and Repository analysis. There I can request a full repository analysis:

I requested repository analysis and was asked whether I wanted code recommendations or code and security. I can see that Code and security involves creating an S3 bucket for the analysis and code and build artifacts are stored there. I did some googling around and it looks like that part is still Java only (but don't quote me on that), so let's just run the code recommendations:

Suggested name is consistent with my lack of talent for consistent naming so I will leave it as it is.

Results

My analysis run returned three issues around resource closing:

The header is a link to the line in GitHub repository, the description of an issue is pretty detailed and links lead to Python docs.

The resources that analysis tool points at are not production code and come from a utility called Versioneer which lets you treat git tags as a project versioning utility, just like Gradle's Axion release plugin which I'm using or universal jgitver. I prefer this solution to editing text files with numbers. As it appears, the code that I was supposed to add to the project for Versioneer to work has some flaws in it.

I've decided to make appropriate corrections: https://github.com/aws-deepracer-community/deepracer-utils/pull/7

The issues have been reviewed and so I have accepted the pull request. No production changes so no release.

Telemetrics

A nice feature that come out of the box is access to telemetrics in CloudWatch:

(The blue dot is showing a number of recommendations found in the repository analysis, and the orange line at zero is showing the amount of issues found in pull requests).

Feedback is one of the substantial elements of a reliable DevOps infrastructure and that applies not only to business, application or system monitoring, but also to developers and quality of their deliverables.

Profiler

A second new feature that I have not looked into is CodeGuru Profiler - it allows you to perform profiling on your Python lambda function or application deployed on EC2. You can have a look at it in the AWS CodeGuru Python support announcement article.

Resources

Apart from the AWS CodeGuru Python support announcement article I recommend that you have a look at AWS CodeGuru Reviewer documentation

As usual, I recommend checking out the pricing page for CodeGuru.

Summary

I'd say the tool looks promising and integrates quite easily into your code. It would be awesome if the tool went a step further and enforced no new issues in pull requests and allowed provision of feedback in the pull request or maybe even the build flow in CodeBuild. One step at a time, I guess.

As any other first-class citizen in the AWS offering (except for DeepRacer, ekhm) AWS CodeGuru offers integration which allows you (among other things) to fetch the recommendations for code reviews so I guess a feedback integration could be scripted. That might be some Fix it Friday project.