반응형
블로그 이미지
개발자로서 현장에서 일하면서 새로 접하는 기술들이나 알게된 정보 등을 정리하기 위한 블로그입니다. 운 좋게 미국에서 큰 회사들의 프로젝트에서 컬설턴트로 일하고 있어서 새로운 기술들을 접할 기회가 많이 있습니다. 미국의 IT 프로젝트에서 사용되는 툴들에 대해 많은 분들과 정보를 공유하고 싶습니다.
솔웅

최근에 올라온 글

최근에 달린 댓글

최근에 받은 트랙백

글 보관함

카테고리

'2019/03'에 해당되는 글 1

  1. 2019.03.19 [ML] AWS ML, AI, SageMaker - With Python lecture (Udemy)


반응형

Udemy - AWS Machine Learning, AI, SageMaker - With Python




Summary


Section 3 Linear Regression


23. Summary


Squared Loss Function is parabolic in nature. It has an important property of not only telling us the loss at a given weight, but also tells us which way to go to minimize loss


Gradient Descent optimization algorithm uses loss function to move the weights of all the features and iteratively adjusts the weights until optimal value is reached


Batch Gradient Descent predicts y value for all training examples and then adjusts the value of weights based on loss. It can converge much slower when training set is very large. Training set order does not matter as every single example in the training set is considered before making adjustments.


Stochastic Gradient Descent predicts y value for next training example and immediately adjusts the value of weights. It can converge faster when training set is very large. Training set should be random order otherwise model will not learn correctly. AWS ML uses stochastic Gradient Descent



Section 4 AWS - Linear Regression Models


27. Concept - How to evaluate regression model accuracy?


Linear Regression - Residuals


- AWS ML Console provides a Histogram that shows distribution of examples that were over estimated and underestimated and to what extent

- Available as "explore model performance" option under Evaluation -> Summary

- Ideal: Over/Under estimation should be a normal curve centered at 0.

- Structural Issue: When you observe vast majority of example falling into one side. Adding more relevant features can help remedy the situation.


31. Model Performance Summary and Conclusion

RMSE (Root Mean Square Error) is the evaluation metric for Linear Regression. Smaller the value of RMSE, better the predictive accuracy of model. Perfect model would have RMSE of 0.


To prepare data for AWS ML, it requires data to be in

1. CSV file available in S3

2. AWS Redshift Datawarehouse

3. AWS Relational Database Service (RDS) MySQL DB


Batch Prediction results are stored by AWS ML to S3 in the specified bucket


We pulled the data from S3 to local folder and plotted them


Based on the distribution of data, AWS ML suggests a recipe for processing data.

In case of numeric features, it may suggest binning the data instead of treating a raw numeric

For this example, treating x as numeric provided best results






Section 5 Adding Features To Improve Model


35. Summary

1. Underfitting occurs when model does not accurately capture relationship between features and target

2. Underfitting would cause large training errors and evaluation errors   

  Training RMSE: 385.1816, Evaluation RMSE: 257.8979, Baseline RMSE: 437.311

3. Evaluation Summary - Prediction overestimation and underestimation histogram provided by AWS ML console provides important clues on how the model is behaving, under-estimation and over-estimation needs to be balanced and centered around 0

4. Box plot also highlights distribution differences between predicted and actual-negatives

5. To address underfitting, add higher order polynomials or more relevant features to capture complex relationship

  Training RMSE: 132.2032, Evaluation RMSE: 63.6847, Baseline RMSE: 437.311

6. When working with datasets containing 100s or even 1000s of features, it important to rely on these metrics and distribution to gain insight into model performance



Section 6 Normalization


37. Concept: Normalization to smoothen magnitude differences


Normalization Transformation (Numeric)

- When there are very large differences in magnitude of features, features that have large magnitude can dominate Model

- Example : We saw this in Quadratic Extra Features dataset

- Normalization is a process of transforming features to have a mean of 0 and variance of 1. This will ensure all features have similar scale

  : Feature normalized = (feature - mean) / (sigma)

    where,

mean = mean of feature x

sigma = standard deviation of feature x

  : Usage : normalize (numericFeature0

- Optimization algorithm may also converge faster with normalized features compared to features that have very large scale differences


39. Summary

1. Having lot of features and complex features can help improve prediction accuracy

2. When feature ranges are orders of magnitude different, it can dominate the outcome. Normalization is a process of transforming features to have a mean of 0 and  variance of 1. This will ensure all feature have similar scale.

3. Without Normalization:

  Training RMSE: 83973.66, Evaluation RMSE: 158260.62, Baseline RMSE: 437.31

4. With Normalization:

  Training RMSE: 72.35, Evaluation RMSE: 51.7387, Baseline RMSE: 437.31

5. Normalization can be easily enabled using AWS ML Transformation recipes



Section 7 Adding Complex Features

46. Summary

Adding polynomial features allows us fit more complex shapes


To add polynomial features that combines all input features, use sci-kit module library. Anaconda includes these modules by default


We saw good performance with degree 4 and any additional feature may bring incremental improvement, but with added complexity of managing features.


1. Model Degree 1 Training RMSE:0.5063, Evaluation RMSE:0.4308, Baseline RMSE:0.689

2. Model Degree 4 Training RMSE:0.2563, Evaluation RMSE:0.1493, Baseline RMSE:0.689

3. Model Degree 15 Training RMSE:0.2984, Evaluation RMSE:0.1222, Baseline RMSE:0.689




Section 8 Kaggle Bike Hourly Rental Prediction


50. Linear Regression Wrapup and Summary


AWS ML - Linear Regression

* Linear Model

* Gradient Descent and Stochastic Gradient Descent

* Squared Error Loss Function

* AWS ML Training, Evaluation, Interactive Prediction, Batch Prediction

* Prediction Quality

  - RMSE

  - Residual Histograms

* Data visualization

* Normalization

* Higher order polynomials



Section 9 - Logistic Regression Models


Image result for Linear vs. Logistic regression model

In short: Linear Regression gives continuous output. i.e. any value between a range of values. ... GLM(Generalized linear models) does not assume a linear relationship between dependent and independent variables. However, it assumes a linear relationship between link function and independent variables in logit model.


https://stackoverflow.com/questions/12146914/what-is-the-difference-between-linear-regression-and-logistic-regression


https://techdifferences.com/difference-between-linear-and-logistic-regression.html


58. Summary

Binary Classifier : Predicts positive class probability of an observation

Logistic or Sigmod function has an important property where output is between 0 and 1 for any input. This output is used by binary classifiers as a probability of positive class.

True Positive - Samples that are actual-positives correctly predicted as positive

True Negative - Samples that are actual-negatives correctly predicted as negative

False Negative - Sampleas that are actual-positives incorrectly predicted as negative

False Positive - Samples that are actual-negatives incorrectly predicted as positive

Logistic Loss Function is parabolic in nature. It has an important property of not only telling us the loss at a given weight. but also tells us which way to go to minimize loss

Gradient Descent optimization algorithm uses loss function to move the weights of all the features and iteratively adjusts the weights until optimal value is reached

Batch Gradient Descent predicts y value for all training examples and then adjusts the value of weights based on loss. It can converge much slower when training set is very large. Training set order does not matter as every single example in the training set is considered before making adjustments.

Stochastic Gradient Descent predicts y value for next training example and immediately adjusts the value of weights. It can converge faster when training set is very large. Training set should be random order otherwise model will not learn correctly. AWS ML uses stochastic Gradient Descent




Section 10 


62

Classification Metrics

True Positive = count(model correctly predicted positives). Students who passed exam correctly classified as pass.

True Negative = count (model correctly predicted negatives). Students who failed exam correctly classfied as fail.

False Positive = count (model misclassified negative as positive). Students who failed exam incorrectly classified as pass.

False Negative = count (model misclassified positive as negative). Students who passes exam incorrectly classified as fail.


* True Positive Rate, Recall, Probability of detection - Fraction of positive predicted correctly. larger value indicates better predictive accuracy.


TPR = True Positive / Actual Positive


* False Positive Rate, probability of false alarm - Fraction of negative predicted as positive. Smaller value indicates better predictive accuracy


FPR = False Positive / Actual Negative


* Precision - Fraction of true positive among all predicted positive. Larger value indicates better predictive accuracy


Precision = True Positive / Predicted Positive


* Accuracy - Fraction of correct predictions. Larger value indicates better predictive accuracy

Accuracy = True Positive + True Negative / negative

where, n is the number of examples


63. 

Classification Insights with AWS Histograms


Histogram - Binary Classifier


* Positive and Negative histograms

* Interactive tool to test effect of various cut-off thresholds

* Ability to save a threshold for the model

* Available under :

Model -> Evaluation Summary -> Explore Performance

https://docs.aws.amazon.com/machine-learning/latest/dg/binary-model-insights.html


64

Concept: AUC Metric


AUC - Binary Classifer

* Area Under Curve(AUC) metric - 0 to 1. Larger Value indicates better predictive accuracy

* AUC is the area of a curve formed by plotting True Positive Rate against False positive Rate at different cut-off thresholds

* AUC value of 0.5 is baseline and it is considered random-guess

* AUC closer to 1 indicates better predictive accuracy

* AUC closer to 0 indicates model has learned correct patterns, but flipping predictions (0's are predicted as 1's and vice versa).


69 Summary

For Binary Classification, Area Under Curve (AUC) is the evaluation metric to assess the quality of model


AUC is the area of a curve formed by plotting True Positive Rate against False Positive Rate at different cut-off thresholds.

* AUC metric closer to 1 indicates highly accurate prediction

* AUC metric 0.5 indicates random guess - Baseline AUC

* AUC metric closer to 0 indicates model has learned from the features, but predictions are flipped


Advanced Metrics

* Accuracy - Fraction of correct predictions. Larger value indicates better predictive accuracy

* True Positive Rate - Probability of detection. Out of all positive, how many were correctly predicted as positive. Larger value indicates better predictive accuracy

* False Positive Rate _ Probability of false alarm. Smaller value indicates better predictive accuracy. Out of all negatives, how many were incorrectly predicted as positive.

* Precision - out of all predicted as positive, how man are true positive? Larger value indicates better predictive accuracy.



Section 11

72 Concept: Evaluating Predictive Quality of Multiclass Classifiers


Multi-class metrics


* F1 Score - Harmonic mean of Recall and Precision. Larger F1 Score indicates better predictive accuracy. Binary Metic


F1 Score = 2.Precision.Recall / Precision + Recall


* Average F1 Score - For multi-class problems, average of class wise F1 score is used for accessing predictive quality


* Baseline F1 Score - Hypothentical model that predicts only most frequent class as the answer 


Concept: Confusion Matrix To Evaluating Predictive Quality


Multiclass - Metrics - Confusion Matrix


* Accessible from Model -> Evaluation Summary -> Explore Model performance


* Concise table that shows percentage and count of correct classification and incorrect classifications


* Visual look at model performance


* Up to 10 classes are shown - listed from most frequent to least frequent


* For more than 10 classes, first 9 most freq. classes are shown and 10th class will collapse rest of the classes and mark as otherwise

* Option to download confusion matrix


* https://docs.aws.amazon.com/machine-learning/latest/dg/multiclass-model-insights.html



77. Summary


Multi-Class Evaluation Metric

1. F1 Score is a binary classification metic. It is harmonic mean of precision and recall

F1 Score = 2 X Precision X Recall / (Precision + Recall)

Higher F1 Score reflects better predictive accuracy

2. Multi-Class Evaluation

Average of class wise F1 Score

3. Baseline F1 Score = Hypothetical model that predicts only most frequent class as the answer

4. Visualization - Confusion Matrix - Available on AWS ML Console

Matrix. Rows = true class. Columns = predicted class

Cell color - diagonal indicates true class prediction %

Cell color - non-diagonal indicates incorrect prediction %

Last column is F1 score for that class. Last but one column is true class distribution

Last row is predicted class distribution

Upto 10 classes are shown - listed from most frequent to least frequent

For more than 10 classes, first 9 most freq. classes are shown and 10th class will collapse rest of the classes and mark as otherwise

You can download the confusion matrix thru url-Explore Performance page under Evaluations


Prediction Summary

1. Eval with default recipe settings. Average F1 score: 0.905

2. Eval with numeric recipe settings: Average F1 score: 0.827

3. Batch prediction Results (predict all 150 example outcome)

  a. With default recipe settings: Average F1 Score: 0.973

  b. With numeric recipe settings:Average F1 Score: 0.78

4. Classification was better with binning. Versicolor classification was impacted when numeric setting was used

5. Higher F1 Score implies better prediction accuracy



Section 12 Text Based Classification with AWS Twitter Dataset


78. AWS Twitter Feed Classification for Customer Service

https://github.com/aws-samples/machine-learning-samples/tree/master/social-media 


79. Lab: Train, Evaluate Model and Assess Predictive Quality, 80. Lab: Interactive Prediction with AWS

- Practice


81. Logistic Regression Summary


AWS ML - Logistic Regression

- Linear Model

- Logistic/Sigmoid Function to produce a probability

- Stochastic Gradient Descent

- Logistic Loss function

- AWS ML Training, Evaluation, Interactive Prediction, Batch Prediction

- Prediction Quality 

  : TPR

  : FPR

  : Accuracy

  :Prediction

  : AUC Metrics

  : F1 Score

  : Average F1 Score for multi-class

- Data visualization

- Text Processing

- Normalization

- Higher order polynomials




Section 13


82. Recipe Overview


Recipe

- Recipe is a set of instructions for pre-processing data

- Recipe is a JSON like document

- Consists of three parts: Groups, Assignments, Outputs

- Groups - Groups are collection of features for which similar transformation needs to be applied

  : Built-in Group : ALL_TEXT, ALL_NUMERIC, ALL_CATEGORICAL, ALL_BINARY

  : Define your own groups

- Assignments - Enable creation of new features derived from existing ones

- Outputs - List features used for learning process and optionally apply transformation


Recipe is automatically applied to training data, evaluation data and to data submitted through real-time and batch prediction APIs


83. Recepe Example


84. Text Transformation


* N-gram Text Transformation

- Tokenizes input text and combines them into a slideing window of n-words, where n is specified in the recipe

- Usage: ngram(textFeature, n), where n is the size

- By default all text data is tokenized with n=1

  : Example: "Customer requests urgent response" text is tokenized as {"Customer", "requests", "urgent", "response"}

- With n=2, it generates one word and two word combinations

  : {"Customer requests", "requests urgent", urgent response", "Customer", "requests", "urgent", "response"}

- N-grams of size up to 10 is supported

- N-grams breaks texts at whitespace. Punctuations will be considered part of word

- You can remove punctuations using no_punct transformation


* OSB Text Transformation

- Orthogonal Spare Bigram (OSB) Transformation provides more word combinations compared to n-gram

- Usage: osb(textFeature, size)

- Puts one underscore to indicate word boundary as well as every word skipped

- For example (AWS Document provided sample).

https://docs.aws.amazon.com/ko_kr/machine-learning/latest/dg/data-transformations-reference.html

Text: "The quick brown fox jumps over the lazy dog". osb(text,4)

WINDOW,{OSB GENERATED}

"The quick brown fox", {The_quick, The__brown, The___fox}

"quick brown fox jumps", {quick_brown, quick__fox, quick___jumps}

"brown fox jumps over", {brown_fox, brown__jumps, brown___over}

"fox jumps over the", {fox_jumps, fox__over, fox___the}

"jumps over the lazy", {jumps_over, jumps__the, jumps___lazy}

"over the lazy dog", {over_the, over__lazy, over___dog}

"the lazy dog", {the_lazy, the__dog}

"lazy dog", {lazy_dog}


* Lowercase and Punctuation


- Lower Case Transformation converts text to lowercase

  : Usage : lowercase(textFeature)

  : Example: "The Quick Brown Fox Jumps Over the Lazy Dog" ->  "the quick brown fox jumps over the lazy dog"

- Remove punctuation Transformation - removes punctuations at word boundaries

  : Usage: nopunct(textFeature)

  : Example: "Customer Number: 123. Ord-No: AB1235" will be by default tokenized as

    {"Customer","Number:","123.","Ord-No:","AB1235"}

  : With nopunct transformation -> {"Customer","Number","123","ord-No","AB1235"}

  : Note: only prefix, suffix punctuations are removed. Embedded punctuations are not removed "Ord-No"

  

85. Numeric Transformation - Quantile Binning


* Quantile Binning Transformation (Numeric)

- Used for converting a numeric value into a categorical bin number

- Usage: quantile_bin(numericFeature, n), where n is the number of bins

- AWS ML uses this information to establish n bins of equal size based on the distribution of all values of the specified numeric feature.

- It then maps incoming numericFeature value to corresponding bin and outputs bin number as categorical value

- AWS ML Recommendation: In some cases, relationship between numeric variable and target is not linear...... binning might be usful in those scenarios

- We actually saw where binning improved predictive accuracy with Iris Dataset


86. Numeric Transformation - Normalization


Normalization Transformation (Numeric)


- When there are very large differences in magnitude of features, features that have large magnitude can dominate Model

- Example: We saw this in Quadratic Extra Features dataset

- Normalization is a process of transforming features to have a mean of 0 and variance of 1. This will enshre all features have similar scale.

  : Example Feature normalized = (feature - mean)/(sigma)

    where,

mean = mean of feature x

sigma = standard deviation of feature x

  : Usage normalize(numericFeature)

- Optimization algorithm may also converge faster with normalized features compared to features that have very large scale differences


87. Cartesian Product Transformation - Categorical and Text


* Cartesian Product Transformation (Categorical, Text)

- Cartesian transformation generates permutations of two or more text and categorical input variables

- For example: Season and Hour combined may have stronger influence on bike rentals. Instead of treating these two as separate features, we can create a new feature Season_Hour that will combine these values.

- Usage cartesian(feature1, feature2)

- Combined features may be able to more accurately related the target attribute

Table


88. Summary

Data Transformation



Section: 14 Hyper Parameters, Model Optimization and Lifecycle

Hyper Parameters allow you to control the model building process and quality


90. Data Rearrangement, Maximum model Size, passes, Shuffle Type

Table



93. Improving Model Quality

Optimizing Model

- To improve a model following are some options

  : Add more training examples

  : Add more relevant features

  : Model hyperparameter tuning

- Quality Metrics of Training Data and Evaluation Data can provide important clues to improve model performance



94. Model Maintenance


- Models may need to be periodically rebuilt or updated to 

  : Keep in-sync with new patterns

  : Support new more relevant features

  : Support new class - in multi - class problems

  : Changes in assumptions or distribution of data that was used to train model

  : Changes to cut-off threshold

  Example: Home price changes month to month depending on several factors

- Have a plan to evaluate model with new data periodically. Example: Weekly, Monthly, Quartly

- Models are probabilistic in nature...

  : Binary Class - Provides bestAnswer(1 or 0) and a raw prediction score. Cut-off score is configurable

  : Multi Class - Provides prediction score for each class. It can be interpreted as probability of observation belonging to the class. Class with highest score is the best answer

  : Regression : Provides a score that contains raw numeric prediction of the target attribute.

- When models are changed, predicted results would also change - Quality metrics like AUC, F1 Score, RMSE can be used to determine whether to go ahead with proposed model change


95. AWS Machine Learning System Limits

- AWS ML imposes certain limits to ensure robust and reliable service

- Some are soft limits and can be increased by contacting AWS Customer Service

- Size of each observation: 100KB

- Size of training data: 100GB

- Size of batch prediction input: 1TB (single file limit. can be overcome by creating more batch files)

- No. of records per batch file: 100 million

- No. of variables/features: 10,000

- Throughput per second for realtime prediction: 200 requests/second

- Max Number of classes per multi-class model: 100


96. AWS Machine Learning Pricing


- Data Analysis and Model Building Fee - $0.42 per Hour of building time

  : Number of computer hours required for data analysis, model training and evaluation

  : Depends on size of input data, attributes, types of transformations applied

- Predictions Fees

  : Batch predictions - $0.10/1,000 predictions founded to the nearest 1,000

  : Real-time predictions - $0.0001 per prediction + Capacity reservation charge of $0.001 per hour for each 10MB provisioned for your model

  

Section 15 Integration of AWS Machine Learning With Your Application


98. Introduction


AWS ML Integration


- Speed!

  : Turn your ideas into cool products in a matter of days

  : Traditional approach would require months

  

- Highly scalable, secure service with redundancy built-in

  : Scale automatically to train model with very large datasets

  : Scale automatically to support high volume prediction needs

  : Real-time prediction with capacity reservation

  : Secure - Limit access to Authenticated and Authorized services and users

  

- Server less!


- Software Integration

  : AWS Machine Learning - Complete functionality is accessible through SDK and Command Line Interfaces

  : Model building and Prediction can be fully automated using SDK

  : AWS SDKs in multiple lanuages - Python, Java, .NET, Javascript, Ruby, C++, ....

  : Complete list languages https://aws.amazon.com/tools/ 

  

99. Integration Scenarios


Connectivity and Security Options


- You Data Center -> AWS ML Cloud Service  

  : Security: Key Based Authentication + IAM Policy + SSL

- AWS Hosted Application -> AWS ML Cloud Service

  : Security : IAM Role + SSL

- Browser, Apps on Phone -> AWS ML Cloud Service

  : Option 1: AWS Cognito Based Authentication + IAM Role + SSL

  : Choice of authentication providers: Cognito, Google, Amazon, Facebook, Twitter, OpenID, Customer

  : Option 2 : Key Based Authentication + IAM Policy + SSL


100. Security using IAM


Users belong to AWS root account. Cognito Users are application level users. Application belongs to AWS root account.




반응형
이전 1 다음