8000 fix: check optional keyword before accessing by chuyang-deng · Pull Request #1911 · aws/sagemaker-python-sdk · GitHub
[go: up one dir, main page]
Skip to content

fix: check optional keyword before accessing #1911

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
ajaykarpur merged 7 commits into from
Oct 7, 2020
Merged

fix: check optional keyword before accessing #1911

ajaykarpur merged 7 commits into from
Oct 7, 2020

Conversation

@chuyang-deng
Copy link
Contributor
@chuyang-deng chuyang-deng commented Sep 21, 2020

Issue #, if available:
#1910

Description of changes:
Check optional Keyword Environment, NetworkConfig before accessing them in ModelMonitor.attach

Testing done:
tox -e py36 -- tests/unit

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

  • I have read the CONTRIBUTING doc
  • I used the commit message format described in CONTRIBUTING
  • I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
  • I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

  • I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • I have checked that my tests are not configured for a specific region or account (if appropriate)
  • I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@sagemaker-bot
Copy link
Collaborator
sagemaker-bot commented Sep 21, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: f3027ca
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator
sagemaker-bot commented Sep 21, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: f3027ca
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator
sagemaker-bot commented Sep 21, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: f3027ca
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator
sagemaker-bot commented Sep 21, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: f3027ca
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@icywang86rui
Copy link
Contributor
icywang86rui commented Sep 21, 2020

This attached function is over 100 lines long now. Should we move some of the dict parsing stuff out or maybe model the DescribeSchedule call output into a class and do all the parsing there?

icywang86rui
icywang86rui reviewed Sep 21, 2020
View reviewed changes
src/sagemaker/model_monitor/model_monitoring.py Outdated
].get("VpcConfig")
vpc_config = None
if (
schedule_desc["MonitoringScheduleConfig"]["MonitoringJobDefinition"].get(
Copy link
Contributor
@icywang86rui icywang86rui Sep 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be just if network_config_dict is not None

Copy link
Contributor
@icywang86rui icywang86rui Sep 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also put all the network_config parsing stuff in the same if block.

Copy link
Contributor Author
@chuyang-deng chuyang-deng Sep 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense! I also refactored the attach method to make it more concise

ajaykarpur
ajaykarpur reviewed Sep 21, 2020
View reviewed changes
tests/unit/sagemaker/monitor/test_model_monitoring.py Outdated
image_uri=CUSTOM_IMAGE_URI,
sagemaker_session=sagemaker_session,
)
model_monitor_schedule_name = "model-monitoring-without-netwotk-config"
Copy link
Contributor
@ajaykarpur ajaykarpur Sep 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
model_monitor_schedule_name = "model-monitoring-without-netwotk-config"
model_monitor_schedule_name = "model-monitoring-without-network-config"

@sagemaker-bot
< 8000 span class="Button-content"> Copy link
Collaborator
sagemaker-bot commented Sep 21, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: 2d53da5
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator
sagemaker-bot commented Sep 21, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: 2d53da5
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator
sagemaker-bot commented Sep 21, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: 2d53da5
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator
sagemaker-bot commented Sep 21, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 2d53da5
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator
sagemaker-bot commented Sep 23, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: 21c0bcb
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator
sagemaker-bot commented Sep 23, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: b0e069b
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator
sagemaker-bot commented Sep 23, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: 21c0bcb
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator
sagemaker-bot commented Sep 23, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: b0e069b
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator
sagemaker-bot commented Sep 23, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: 21c0bcb
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator
sagemaker-bot commented Sep 23, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: b0e069b
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator
sagemaker-bot commented Sep 23, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: b0e069b
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator
sagemaker-bot commented Sep 24, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: af618c1
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator
sagemaker-bot commented Sep 24, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: af618c1
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator
sagemaker-bot commented Sep 24, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: af618c1
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator
sagemaker-bot commented Sep 24, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: af618c1
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator
sagemaker-bot commented Oct 7, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: 0b5032c
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator
sagemaker-bot commented Oct 7, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 0b5032c
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator
sagemaker-bot commented Oct 7, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: 0b5032c
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator
sagemaker-bot commented Oct 7, 2020

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: 0b5032c
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@ajaykarpur ajaykarpur merged commit 2b812c8 into aws:master Oct 7, 2020
@ajaykarpur ajaykarpur mentioned this pull request Oct 7, 2020
Closed
guoqiao1992 pushed a commit to guoqiao1992/sagemaker-python-sdk that referenced this pull request Oct 26, 2020
@chuyang-deng @icywang86rui
@ajaykarpur @guoqiao1992
fix: check optional keyword before accessing (aws#1911)
daba222 
* fix: check optional keyword before accessing

* refactor attach method

Co-authored-by: Chuyang Deng <chuyangd@amazon.com>
Co-authored-by: icywang86rui <icywang.rui@gmail.com>
Co-authored-by: Ajay Karpur <akarpur@amazon.com>
rsareddy0329 added a commit that referenced this pull request Dec 3, 2025
@mohamedzeidan2021 @mollyheamazon
@jam-jee @rsareddy0329 @nargokul @mufaddal-rohawala @mufiAmazon @zhaoqizqwang
Master model customization to keynote 3 v3 (#1925)
53a3f2d 
* sagemaker-core update, port sft-trainer and ai registry (#1895)

* sagemaker core update with MC service.json, SFT notebook flow succeeded before modelpackage

* remove workaround, modelpackage issue persists

* remove ai registry port

* feat: AIRegistry SDK Implementation (#1893)

* remove sft trainer port, fix ModelPackage issue (#1901)

* Add/port finetuning interfaces to v3 (#1906)

* Add/port finetuning interfaces to v3

* Add/port finetuning interfaces to v3

---------

Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>

* ported evaluation from v2->v3. pending example notebooks results (#1902)

* v2->v3 porting for eval

* bring back deleted needed files

* unit tests

---------

Co-authored-by: Mohamed Zeidan <zeidmo@amazon.com>

* Mc deployment (#1909)

* Updates for Model Customization

* Model Builder Base change

* Fix

* Support for Model Customization

* New Model Builder changes with Model Customization

* Fix

* Add base_trainer, addition to PR#1906 (#1910)

* Add/port finetuning interfaces to v3

* Add/port finetuning interfaces to v3

* Add base_trainer, addition to PR#1906

---------

Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>

* AIRegistry Integration Tests (#1908)

* feat: Trainer wait with MLFlow metrics (#1907)

* feat: update eval integs (#1912)

Co-authored-by: Mufaddal Rohawala <mufi@amazon.com>

* Update get_all method for Evaluator and Dataset (#1913)

* Update get_all method for Evaluator and Dataset

- Update Dataset.get_all() and added refresh()
- Update Evaluator.get_all() and added refresh()
- Added more tags in condition when importing hub content

* Update dataset, evaluators and examples

* remove hub_content_name

* port bedrock model builder and add integ test (#1911)

* port bedrock model builder and add integ test

* Add unit test and notebook, port new version of bedrock model builder

* add docstring

* Mc deployment integ tests and example notebook (#1914)

* Updates for Model Customization

* Model Builder Base change

* Fix

* Support for Model Customization

* New Model Builder changes with Model Customization

* Fix

* Deployment Integ tests

* Tests

* Deployment NOtebook

* fix

* Update model hub name and encoding method (#1918)

* Update get_all method for Evaluator and Dataset

- Update Dataset.get_all() and added refresh()
- Update Evaluator.get_all() and added refresh()
- Added more tags in condition when importing hub content

* Update dataset, evaluators and examples

* remove hub_content_name

* Update hub name and hash

- Update model hub name to be AIRegistry-{accountId}-{region}
- Update hub name hash

* Add telemetry for model customization (#1920)

* add telemetry to model customization

* add telemetry to model customization 2

* small fix

* INteg tests for nova bedrock  (#1926)

* Bedrock Nova

* Unit tests for bedrock nova

* accept eula always (#1928)

Co-authored-by: Mohamed Zeidan <zeidmo@amazon.com>

* Add studio tags to trainer (#1927)

* Update get_all method for Evaluator and Dataset

- Update Dataset.get_all() and added refresh()
- Update Evaluator.get_all() and added refresh()
- Added more tags in condition when importing hub content

* Update dataset, evaluators and examples

* remove hub_content_name

* Update hub name and hash

- Update model hub name to be AIRegistry-{accountId}-{region}
- Update hub name hash

* Add studio tags to trainer

Tested nu checking training job creation inputs and unit tests

---------

Co-authored-by: rsareddy0329 <rsareddy0329@gmail.com>

* Finetuning classes: add accept_eula support & Add integ tests for sft, rlvr,rlaif, dpo trainers (#1915)

* Add/port finetuning interfaces to v3

* Add/port finetuning interfaces to v3

* Add base_trainer, addition to PR#1906

* Add integ tests for sft, rlvr,rlaif, dpo trainers

* Add integ tests for sft, rlvr,rlaif, dpo trainers

* Finetuning classes: Add accept_eula support

* Finetuning classes: Add accept_eula support

---------

Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>

* feat: pipeline name fix + mlflow arn fix + jinja dep (#1931)

* changes for pipeline naming

* feat: pipeline name fix + mlflow arn fix + jinja dep

---------

Co-authored-by: Mufaddal Rohawala <mufi@amazon.com>

* AIRegistry: Add Support to user provided session and role (#1932)

* Add/port finetuning interfaces to v3

* Add/port finetuning interfaces to v3

* Add base_trainer, addition to PR#1906

* Add integ tests for sft, rlvr,rlaif, dpo trainers

* Add integ tests for sft, rlvr,rlaif, dpo trainers

* Finetuning classes: Add accept_eula support

* Finetuning classes: Add accept_eula support

* AIRegistry: Support user provided session and role

* AIRegistry: Support user provided session and role

* AIRegistry: Support user provided session and role

---------

Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>

* format name for ai reg, stripped name of =, added domain-id tags for … (#1930)

* format name for ai reg, stripped name of =, added domain-id tags for datasets and evaluators, prefix changed t use full path including filename

* pulling metadata path since get_current_domain_id didnt work

* rmvd gamma endpoint

---------

Co-authored-by: Mohamed Zeidan <zeidmo@amazon.com>

* fixed train unit tests (#1933)

Co-authored-by: Mohamed Zeidan <zeidmo@amazon.com>

* Updating DataSet versioning pattern for parity with UI (#1934)

* fix dataset arn configuration in eval (#1935)

Co-authored-by: Mufaddal Rohawala <mufi@amazon.com>

* Master model customization v3 (#1936)

* Updating DataSet versioning pattern for parity with UI

* Updating default S3 bucket and prefix for datasets

* fix: custom metrics and pipeline name in lineage (#1937)

* remove mlflow config for base model eval only

* fix unit

* fix custom metrics and pipeline name in lineage

---------

Testing:
1. Added/Ran units.
2. Tested LLMAJ eval.

Co-authored-by: Mufaddal Rohawala <mufi@amazon.com>

* Integ tests for ModelBuilder (#1939)

* Bedrock Nova

* Unit tests for bedrock nova

* Integ test updates

* Test case updates

* fixing AIRegistry integration and unit tests (#1941)

* Update error messages based on feedback (#1938)

* feat: Inline MLFlow metrics with Serverless MLFlow App (#1942)

* Master model customization v3 (#1945)

* Bedrock Nova

* Unit tests for bedrock nova

* Integ test updates

* Test case updates

* Bedrock tests

* fix: update eval integ tests (#1946)

* update eval integ tests

* fix unit tests

---------

Co-authored-by: Mufaddal Rohawala <mufi@amazon.com>

* Finetuning classes: Add integ tests (#1947)

* Add/port finetuning interfaces to v3

* Add/port finetuning interfaces to v3

* Add base_trainer, addition to PR#1906

* Add integ tests for sft, rlvr,rlaif, dpo trainers

* Add integ tests 
8212
for sft, rlvr,rlaif, dpo trainers

* Finetuning classes: Add accept_eula support

* Finetuning classes: Add accept_eula support

* AIRegistry: Support user provided session and role

* AIRegistry: Support user provided session and role

* AIRegistry: Support user provided session and role

* Finetuning classes: Add integration tests

* Merge conflicts

* Merge conflicts

---------

Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>

* Workflow change to enable github actions (#1948)

* workflow change and test

* remove test

* Remove beta endpoint info, add region validation and tests  (#1950)

* Add/port finetuning interfaces to v3

* Add/port finetuning interfaces to v3

* Add base_trainer, addition to PR#1906

* Add integ tests for sft, rlvr,rlaif, dpo trainers

* Add integ tests for sft, rlvr,rlaif, dpo trainers

* Finetuning classes: Add accept_eula support

* Finetuning classes: Add accept_eula support

* AIRegistry: Support user provided session and role

* AIRegistry: Support user provided session and role

* AIRegistry: Support user provided session and role

* Finetuning classes: Add integration tests

* Merge conflicts

* Merge conflicts

* Remove beta endpoint details in sagemaker-core

* add region validation for models

---------

Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>

---------

Co-authored-by: Molly He <mollyhe@amazon.com>
Co-authored-by: jam-jee <jamjee@amazon.com>
Co-authored-by: rsareddy0329 <rsareddy0329@gmail.com>
Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>
Co-authored-by: Mohamed Zeidan <zeidmo@amazon.com>
Co-authored-by: Gokul Anantha Narayanan <166456257+nargokul@users.noreply.github.com>
Co-authored-by: Mufaddal Rohawala <89424143+mufaddal-rohawala@users.noreply.github.com>
Co-authored-by: Mufaddal Rohawala <mufi@amazon.com>
Co-authored-by: Zhaoqi <52220743+zhaoqizqwang@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

2 more reviewers

@icywang86rui icywang86rui icywang86rui left review comments

@ajaykarpur ajaykarpur ajaykarpur approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@chuyang-deng @sagemaker-bot @icywang86rui @ajaykarpur
0