반응형
블로그 이미지
개발자로서 현장에서 일하면서 새로 접하는 기술들이나 알게된 정보 등을 정리하기 위한 블로그입니다. 운 좋게 미국에서 큰 회사들의 프로젝트에서 컬설턴트로 일하고 있어서 새로운 기술들을 접할 기회가 많이 있습니다. 미국의 IT 프로젝트에서 사용되는 툴들에 대해 많은 분들과 정보를 공유하고 싶습니다.
솔웅

최근에 올라온 글

최근에 달린 댓글

최근에 받은 트랙백

글 보관함

카테고리

[AWS Certificate] Developer - Route53 memo

2017. 11. 25. 08:37 | Posted by 솔웅


반응형

Route53 & DNS



What is DNS?


If you've used the internet, you've used DNS. DNS is used to convert human friendly domain names (such as http://acloud.guru) into an Internet Protocol (IP) address (such as http://82.124.53.1).


IP addresses are used by computers to identify each other on the network. IP addresses commonly come in 2 different forms, IPv4 and IPv6.



IPv4 vs. IPv6


The IPv4 space is a 32 bit field and has over 4 billion different addresses (4,294,967,296 to be precise).


IPv6 was created to solve this depletion issue and has an address space of 128 bits which in theory is

340,282,366,920,938,463,463,374,607,431,768,211,456 addresses or 340 undecillion addresses





Top Level Domains


If we look at common domain names such as google.com, bbc.co.uk. acloud,guru etc. you will notice a string of characters separated by dots (periods). The last word in a domain name represents the "top level domain". The second word in a domain name is known as a second level domain name (this is optional though and depends on the domain name).

.com, .edu, .gov, .co.uk, .gov.uk, .com.au


These top level domain names are controlled by the Internet Assigned Numbers Authority (IANA) in a root zone database which is essentially a database of all available top level domains. You can view this database by visiting

http://www.iana.org/domains/root/db







Domain Registrars



Because all of the names in a given domain name have to be unique there needs to be a way to organize this all so that domain names aren't duplicated. This is where domain registrars come in. A registrar is an authority that can assign domain names directly under one or more top-level domains. These domains are registered with InterNIC, a service of ICANN, which enforces uniqueness of domain names across the Internet. Each domain name becomes registered in a central database known as the WhoIS database.


Popular domain registrars include GoDaddy.com, 123-reg.co.uk etc.



SOA Records


The SOA record stores information about


- The name of the server that supplied the data for the zone.

- The administrator of the zone.

- The current version of the data file.

- The number of seconds a secondary name server should wait before checking for updates

- The number of seconds a secondary name server should wait before retrying a failed zone transfer

- The maximum number of seconds that a secondary name server can use data before it must either be refreshed or expire.

- The default number of seconds for the time-to-live file on resource records.



NS Records


NS stands for Name Server records and are used by Top Level Domain servers to direct traffic to the Content DNS server which contains the authoritative DNS records.



A Records


An 'A' record is the fundamental type of DNS record and the 'A' in A record stands for 'Address'. The A record is used by a computer to translate the name of the domain to the IP address. For example http://www.acloud.guru might point to http://123.10.10.80



TTL


The length that a DNS record is cached on either the Resolving Server or the users own local PC is equal to the value of the "Time To Live" (TTL) in seconds. The lower the time to live, the faster changes to DNS records take to propagate throughout the internet.



CNAMES


A Canonical Name (CName) can be used to resolve one domain name to another. For example, you may have a mobile website with the domain name http://m.acloud.guru that is used for when users browse to your domain name on their mobile devices. You may also want the name http://mobile.acloud.guru to resolve to this same address.



Alias Records


Alias records are used to map resource record sets in your hosted zone to Elastic Load Balancers, CloudFront distributions, or S3 buckets that are configured as websites.


Alias records work like a CNAME record in that you can map one DNS name (www.example.com) to another 'target' DNS name (elb1234.elb.amazonaws.com).


Key difference - A CNAME can't be used for naked domain names (zone apex record). You can't have a CNAME for http://acloud.guru, it must be either an A record or an Alias.


Alias resource record sets can save you time because Amazon Route 53 automatically recognizes changes in the record sets that the alias resource record set refers to.


For example, suppose an alias resource record set for example.com points to an ELB load balancer at lb1-1234.us-east-1.elb.amazonaws.com. If the IP address of the load balancer changes, Amazon Route 53 will automatically reflect those changes in DNS answers for example.com without any changes to the hosted zone that contains resource record sets for example.com.





Exam Tips


- ELB's do not have pre-defined IPv4 addresses, you resolve to them using a DNS name.

- Understand the difference between an Alias Record and a CNAME.

- Given the choice, always choose an Alias Record over a CNAME


==================================


Route 53 - Register A Domain Name 


AWS Console - Networking - Route 53 - Registered Domains - Register New Domain - 



=====================================


Set up EC2 Instances


Set up 2 Instances - create html files

Set up LoadBalancer - DNS name -> will display html file in 1 of 2 Instances


Change Region

Setup an Instance - create html files - Create new security group - Create new key - Launch

Create new Region ELB


DNS name - display html file in new Region instance


=================================


Simple Routing Policy Lab


- Simple

This is the default routing policy when you create a new record set. This is most commonly used when you have a single resource that performs a given function for your domain, for example, one web server that serves content for the http://acloud.guru website.





AWS Console - Route53 - Create Hosted Zone - click on DNS link - Create Record Set

-> Alias Target - ELB


=========================


- Weighted Routing Policy


Weighted Routing Policies let you split your traffic based on different weights assigned. For example you can set 10% of your traffic to go to US-EAST-1 and 90% to go to EU-WEST-1.


AWS Console - Route 53 - Create Record Set - Alias - Select ELB - Routing Policy : Weighted - Enter Weight (90%) and Set ID - Click on Create Button


Create Record Set - Select other ELB - Enter Weight (10%)





==========================


Latency Routing Policy


Latency based routing allows you to route your traffic based on the lowest network latency for your end user (i.e. which region will give them the fastest response time).


To use latency-based routing you create a latency resource record set for the Amazon EC2 (or ELB) resource in each region that hosts your website. When Amazon Route 53 receives a query for your site, it selects the latency resource record set for the region that gives the user the lowest latency. Route 53 then responds with the value associated with that resource record set.


AWS Console - Route 53 - Create Record Set - Alias Target (ELB) - Routing Policy (Latency) - Set ID - Select Region 1


AWS Console - Route 53 - Create Record Set - Alias Target (ELB) - Routing Policy (Latency) - Set ID - Select Region 2




==========================


Failover Routing Policy



Failover routing policies are used when you want to create an active/passive set up. For example you may want your primary site to be in EU-WEST-2 and your secondary DR Site in AP-SOUTHEAST-2.


Route 53 will monitor the health of your primary site using a health check.


A health check monitors the health of your end points.


AWS Console - ELB : Copy DNS name - Route 53 - Health check - Name 1, Domain Name, enter advanced configuration - Create health check


AWS Console - ELB : Copy DNS name - Route 53 - Health check - Name 2, Domain Name, enter advanced configuration - Set Alarm : Set SNS Topic - Create health check


AWS Console - Route 53 - Create Record Set - Alias Target (ELB) - Routing Policy : Failover, Set Primary or Secondary, Set Associate with Health Check 


AWS Console - Route 53 - Create Record Set - Alias Target (ELB) - Routing Policy : Failover, Set Primary or Secondary



==========================



Geolocation Routing Policy



Geolocation routing lets you choose where your traffic will be sent based on the geographic location of our users (i.e. the location from which DNS queries originate). For example, you might want all queries from Europe to be routed to a fleet of EC2 instances that are specifically configured for your European customers. These servers may have the local language of your European customers and all prices are displayed in Euros.


AWS Console - Route 53 - Create Record Set - Alias (ELB) - Routing Policy : Geolocation - US or Europe etc. , Set ID


AWS Console - Route 53 - Create Record Set - Alias (ELB) - Routing Policy : Geolocation - US or Europe etc. , Set ID




===========================


DNS Summary


DNS Exam Tips


Delete all Load balancers. It is paid service.


ELB has no IP address - only DNS name


- ELB's do not have pre-defined IPv4 addresses, you resolve to them using a DNS name.

- Understand the difference between an Alias Record and a CNAME.

- Given the choice, always choose an Alias Record over a CNAME.

- Remember the different routing policies and their use cases.

: Simple

: Wighted

: Latency

: Failover

: Geolocation



http://realmojo.tistory.com/179





반응형


반응형

CloudFormation



What is CloudFormation?




One of the most powerful parts of AWS, CloudFormation allows you to take what was once traditional hardware infrastructure and convert it into code.


CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion.


You don't need to figure our the order for provisioning AWS services or the subtleties of making those dependencies work. CloudFormation takes care of this for you.


After the AWS resources are deployed, you can modify and update them in a controlled and predictable way, in effect applying version control to your AWS infrastructure the same way you do with your software.




CloudFormation Stack vs. Template


A CloudFormation Template is essentially an architectural diagram and a CloudFormation Stack is the end result of that diagram (i.e. what is actually provisioned).


You create, update, and delete a collection of resources by creating, updating, and deleting stacks using CloudFormation templates.


CloudFormation templates are in the JSON format or YAML.



Elements of A Template


Mandatory Elements

- List of AWS Resources and their associated configuration values


Optional Elements

- The template's file format & version number

- Template Parameters

  : The input values that are supplied at stack creation time. Limit of 60

- Output Values

  : The output values required once a stack has finished building (such as the public IP address, ELB address, etc.) Limit of 60.

- List of data tables

  : Used to look up static configuration values such AMI's etc.

  


Outputting Data


- You can use Fn:GetAtt to output data



Exam Tips


- By default, the "automatic rollback on error" feature is enabled

- You are charged for errors

- CloudFormation is free

- Stacks can wait for applications to be provisioned using the "WaitCondition"

- You can use Fn:GetAtt to output data

- Route53 is completely supported. This includes creating new hosted zones or updating existing ones.

- You can create A Records, Aliases etc.

- IAM Role Creation and Assignment is also supported.


1~2 questions in Exam


===========================



Cloud Formation Quiz


- The default scripting language for CloudFormation is : JSON

- Cloud Formation itself is free, however the resources it provisions will be charged at the usual rates. : True

- What happens if Cloud Formation encounters an error by default?

  : It will terminate and rollback all resources created on failure

- You are creating a virtual data center using cloud formation and you need to output the DNS name of your load balancer. What command would you use to achieve this?

  : FN::GetAtt

- What language are cloud formation templates written in? : JSON



======================================


Shared Responsibility Model



===========================


Shared Responsibility Model Quiz


- You are required to patch OS and Applications in RDS? : False

- In the shared responsibility model, what is AWS's responsibility?

  : Restricting access to the data centers, proper destruction of decommissioned disks, patching of firmware for the hardware on which your AWS resources reside



================================




DNS


What is DNS?


If you've used the internet, you've used DNS. DNS is used to convert human friendly domain names (such as http://acloud.guru) into an Internet Protocol (IP) address (such as http://82.124.53.1).


IP addresses are used by computers to identify each other on the network. IP addresses commonly come in 2 different forms, IPv4 and IPv6.



IPv4 vs. IPv6


The IPv4 space is a 32 bit field and has over 4 billion different addresses (4,294,967,296 to be precise).


IPv6 was created to solve this depletion issue and has an address space of 128 bits which in theory is

340,282,366,920,938,463,463,374,607,431,768,211,456 addresses or 340 undecillion addresses



Top Level Domains


If we look at common domain names such as google.com, bbc.co.uk. acloud,guru etc. you will notice a string of characters separated by dots (periods). The last word in a domain name represents the "top level domain". The second word in a domain name is known as a second level domain name (this is optional though and depends on the domain name).

.com, .edu, .gov, .co.uk, .gov.uk, .com.au


These top level domain names are controlled by the Internet Assigned Numbers Authority (IANA) in a root zone database which is essentially a database of all available top level domains. You can view this database by visiting

http://www.iana.org/domains/root/db



Domain Registrars


Because all of the names in a given domain name have to be unique there needs to be a way to organize this all so that domain names aren't duplicated. This is where domain registrars come in. A registrar is an authority that can assign domain names directly under one or more top-level domains. These domains are registered with InterNIC, a service of ICANN, which enforces uniqueness of domain names across the Internet. Each domain name becomes registered in a central database known as the WhoIS database.


Popular domain registrars include GoDaddy.com, 123-reg.co.uk etc.




SOA Records


The SOA record stores information about


- The name of the server that supplied the data for the zone.

- The administrator of the zone.

- The current version of the data file.

- The number of seconds a secondary name server should wait before checking for updates

- The number of seconds a secondary name server should wait before retrying a failed zone transfer

- The maximum number of seconds that a secondary name server can use data before it must either be refreshed or expire.

- The default number of seconds for the time-to-live file on resource records.



NS Records


NS stands for Name Server records and are used by Top Level Domain servers to direct traffic to the Content DNS server which contains the authoritative DNS records.



A Records


An 'A' record is the fundamental type of DNS record and the 'A' in A record stands for 'Address'. The A record is used by a computer to translate the name of the domain to the IP address. For example http://www.acloud.guru might point to http://123.10.10.80



TTL


The length that a DNS record is cached on either the Resolving Server or the users own local PC is equal to the value of the "Time To Live" (TTL) in seconds. The lower the time to live, the faster changes to DNS records take to propagate throughout the internet.



CNAMES


A Canonical Name (CName) can be used to resolve one domain name to another. For example, you may have a mobile website with the domain name http://m.acloud.guru that is used for when users browse to your domain name on their mobile devices. You may also want the name http://mobile.acloud.guru to resolve to this same address.



Alias Records


Alias records are used to map resource record sets in your hosted zone to Elastic Load Balancers, CloudFront distributions, or S3 buckets that are configured as websites.


Alias records work like a CNAME record in that you can map one DNS name (www.example.com) to another 'target' DNS name (elb1234.elb.amazonaws.com).


Key difference - A CNAME can't be used for naked domain names (zone apex record). You can't have a CNAME for http://acloud.guru, it must be either an A record or an Alias.


Alias resource record sets can save you time because Amazon Route 53 automatically recognizes changes in the record sets that the alias resource record set refers to.


For example, suppose an alias resource record set for example.com points to an ELB load balancer at lb1-1234.us-east-1.elb.amazonaws.com. If the IP address of the load balancer changes, Amazon Route 53 will automatically reflect those changes in DNS answers for example.com without any changes to the hosted zone that contains resource record sets for example.com.



Exam Tips


- ELB's do not have pre-defined IPv4 addresses, you resolve to them using a DNS name.

- Understand the difference between an Alias Record and a CNAME.

- Given the choice, always choose an Alias Record over a CNAME





반응형


반응형


SNS (Simple Notification Service)







Amazon Simple Notification Service (Amazon SNS) is a web service that makes it easy to set up, operate, and send notifications from the cloud.


It provides developers with a highly scalable, flexible, and cost-effective capability to publish messages from an application and immediately deliver them to subscribers or other applications.


Amazon SNS follows the "publish-subscribe" (pub-sub) messaging paradigm, with notifications being delivered to clients using a "push" mechanism that eliminates the need to periodically check or "poll" for new information and updates.


With simple APIs requiring minimal up-front development effort, no maintenance or management overhead and pay-as-you-go pricing, Amazon SNS gives developers an easy mechanism to incorporate a powerful notification system with their applications.


Push notifications to Apple, Google, Fire OS, and Windows devices, as well as Android devices in China with Baidu Cloud Push.


Besides pushing cloud notifications directly to mobile devices, Amazon SNS can also deliver notifications by SMS text message or email, to Amazon Simple Queue Service (SQS) queues, or to any HTTP endpoint.


To prevent messages from being lost, all messages published to Amazon SNS are stored redundantly across multiple availability zones.





SNS - Topics



SNS allows you to group multiple recipients using topics. A topic is an "access point" for allowing recipients to dynamically subscribe for identical copies of the same notification.


One topic can support deliveries to multiple endpoint types -- for example, you can group together iOS, Android and SMS recipients. When you publish once to a topic, SNS delivers appropriately formatted copies of your message to each subscriber.






SNS Benefits


- Instantaneous, push-based delivery (no polling)

- Simple APIs and easy integration with applications

- Flexible message delivery over multiple transport protocols

- Inexpensive, pay-as-you-go model with no up-front costs

- Web-based AWS Management Console offers the simplicity of a point-and-click interface




SNS vs. SQS


- Both Messaging Services in AWS

- SNS - Push

- SQS - Polls (Pulls)









SNS Pricing


- Users pay $0.50 per 1 million Amazon SNS Requests

- $0.06 per 100,000 Notification deliveries over HTTP

- $0.75 per 100 Notification deliveries over SMS

- $2.00 per 100,000 Notification deliveries over Email



SNS FAQ



==============


Creating SNS Topic





================



SNS Summary


- Instantaneous, push-based delivery (no polling)

- Protocols include

  : HTTP

  : HTTPS

  : Email

  : Email-JSON

  : Amazon SQS

  : Application

- Messages can be customized for each protocol



====================


SNS Quiz


- SNS is pull based rather than push based? : False

- Which of these is a protocol NOT supported by SNS

  HTTP, HTTPS, Email, Email-JSON, FTP, SQS, Application

  ==> The answer is FTP

- Messages cannot be customized for each protocol used in SNS? : False

- You have a list of subscribers email addresses that you need to push emails out to on a periodic bases. What do you subscribe them to? : A Topic

- You can use SNS in conjunction with SQS to fan a single message out to multiple SQS queues. : True





======================




AWS SWF (Simple Workflow Service)



Amazon Simple Workflow Service (Amazon SWF) is a web service that makes it easy to coordinate work across distributed application components. Amazon SWF enables applications for a range of use cases, including media processing, web application back-ends, business process workflows, and analytics pipelines, to be designed as a coordination of tasks.


Tasks represent invocations of various processing steps in an application which can be performed by executable code, web service calls, human actions, and scripts.



SWF Workers


Workers are programs that interact with Amazon SWF to get tasks, process received tasks, and return the results.



SWF Decider


The decider is a program that controls the coordination of tasks, i.e. their ordering, concurrency, and scheduling according to the application logic.






SWF Workers & Deciders


The workers and the decider can run on cloud infrastructure, such as Amazon EC2, or on machines behind firewalls. Amazon SWF brokers the interactions between workers and the decider. It allows the decider to get consistent views into the progress of tasks and to initiate new tasks in an ongoing manner.


At the same time, Amazon SWF stores tasks, assigns them to workers when they are ready, and monitors their progress. It ensures that a task is assigned only once and is never duplicated. Since Amazon SWF maintains the application's state durably, workers and deciders don't have to keep track of execution state. They can run independently, and scale quickly.




SWF Domains





Your workflow and activity types and the workflow execution itself are all scoped to a domain. Domains isolate a set of types, executions, and task lists from others within the same account.


You can register a domain by using the AWS Management Console or by using the RegisterDomain action in the Amazon SWF API.



The parameters are specified in JavaScript Object Notation (JSON) format.



How Long For workflow?


Maximum Workflow can be 1 year and the value is always measured in seconds.



SWF FAQ



SWF vs SQS


- Amazon SWF presents a task-oriented API, whereas 

  Amazon SQS offers a message-oriented API.

- Amazon SWF ensure that a task is assigned only once and is never duplicated. With Amazon SQS, you need to handle duplicated messages and may also need to ensure that a message is processed only once.

- Amazon SWF keeps track of all the tasks and events in an application. With Amazon SQS, you need to implement your own application-level tracking, especially if your application uses multiple queue.





===========================


SWF Quiz


- SWF consists of a domain, workers an deciders? : True

- Maintaining your application's execution state (e.g. which steps have completed, which ones are running, etc.) is a perfect use case for SWF. : True

- Amazon SWF is useful for automating workflows that include long-running human task (e.g. approvals, reviews, investigations, etc.) Amazon SWF reliably tracks the status of processing steps that run up to several days or months. : True

- In Amazon SWF what is a worker? 

  : Workers are programs that interact with Amazon SWF to get tasks, process received tasks, and return the results

- In Amazon SWF what is a decider

  : The decider is a program that controls the coordination of tasks, i.e. their ordering, concurrency, and scheduling according to the application logic.

  




  

============




Elastic Beanstalk (*** 4~5 questions in the Exam)







- With Elastic Beanstalk, you can deploy, monitor, and scale an application quickly

- It provides developers or end users with the ability to provision application infrastructure is an almost transparent way.

- It has a highly abstract focus towards infrastructure, focusing on components and performance - not configuration and specifications

- It attempts to remove, or significantly simplify infrastructure management, allowing applications to deployed into infrastructure environments easily.





Beanstalk key architecture components


- Applications are the high level structure in beanstalk

- Either your entire application, is one EB application, or

- Each logical component of your application, can be a EB application or a EB environment within an application


- Applications can have multiple environments (Prod, Staging, Dev, V1, V2, V1.1 or functional type (front-end, back-end)

- Environments are either single instance or scalable

- Environments are either web server environments or worker environments


- Application Versions are unique packages which represent versions of apps.

- An application is uploaded to Elastic beanstalk as a application bundle - .zip

- Each application can have many versions 1:M relationship

- Application versions can be deployed to environments within an Application




Elastic Beanstalk Exam Tips


- You can have multiple versions of your applications

- Your applications can be split in to tiers (Web Tier/Application Tier/Database Tier)

- You can update your application

- You can update your configuration

- Updates can be 1 instance at a time, a % of instances or an immutable update

- You pay for the resources that you use, but Elastic Beanstalk is free

- If elastic beanstalk creates your RDS database then it will delete it when you delete your application. If not then the RDS instance stays 

- Know what languages are supported


- Apache Tomcat for Java application

- Apache HTTP Server for PHP applications

- Apache HTTP Server for Python applications

- Nginx or Apache HTTP Server for Node.js applications

- Passenger or Puma for Ruby applications 

- Microsoft IIS 7.5, 8.0, and 8.5 for .NET applications

- JAVA SE

- Docker

- Go




==============================


Elastic Beanstalk Quiz


- Elastic Beanstalk is object based storage. : False

- What languages and development stacks is NOT supported by AWS Elastic Beanstalk?

  : Jetty for jbos application 

- Unlike Cloud Formation, Elastic Beanstalk itself is not free AND you must also pay for the resources it provisions. : False




Elastic Beanstalk FAQ



=====================================



반응형


반응형

Simple Queue Service (SQS) ***






Amazon SQS is a web service that gives you access to a message queue that can be used to store messages while waiting for a computer to process them.

Amazon SQS is a distributed queue system that enables web service applications to quickly and reliably queue messages that one component in the application generates to be consumed by another component. A queue is a temporary repository for messages that are awaiting processing.



Using Amazon SQS, you can decouple the components of an application so they run independently, with Amazon SQS easing message management between components. Any component of a distributed application can store messages in a fail-safe queue.



Messages can contain up to 256 KB (***) of text in any format. Any component can later retrieve the messages programmatically using the Amazon SQS API.



The queue acts as a buffer between the component producing and saving data, and the component receiving the data for processing.




This means the queue resolves issues that arise if the producer is producing work faster than the consumer can process it, or if the producer or consumer are only intermittently connected to the network.



Amazon SQL ensures delivery of each message at least once, and supports multiple readers and writers interacting with the same queue.



A single queue can be used simultaneously by many distributed application components, with no need for those components to coordinate with each other to share the queue.



Amazon SQS is engineered to always be available and deliver messages. One of the resulting tradeoffs is that SQS does not guarantee first in, first out delivery of messages. For many distributed applications, each message can stand on its own, and as long as all messages are delivered, the order is not important.



If your system requires that order be preserved, you can place sequencing information in each message, so that you can reorder the messages when the queue returns them.



To illustrate, suppose you have a number of image files to encode. In an Amazon SQS worker queue, you create an Amazon SQS message for each file specifying the command (jpeg-encode) and the location of the file in Amazon S3.



A pool of Amazon EC2 instances running the needed image processing software does the following





SQS Exam Tips


1. Asynchronously pulls the task messages from the queue

2. Retrieves the named file

3. Processes the conversion

4. Write the image back to Amazon S3

5. Writes a "task complete" message to another queue

6. Delete the original task message

7. Checks for more messages in the worker queue




Autoscaling






- Does not offer FIFO

- 12 hours visibility time out

- Amazon SQS is engineered to provide "at least once" delivery of all messages in its queues. Although most of the time each message will be delivered to your application exactly once, you should design your system so that processing a message more than once does not create any errors or inconsistencies.

- 256kb message size now available

- Billed at 64 kb "chunks"

- A 256kb message will be 4 X 64kb "chunks"




SQL Pricing


- First 1 million Amazon SQS Requests per month are free

- $0.50 per 1 million Amazon SQS Requests per month thereafter ($0.00000050 per SQS Request)

- A single request can have from 1 to 10 messages, up to a maximum total payload of 256KB.

- Each 64KB 'chunk' of payload is billed as 1 request. For example, a single API call with a 256KB payload will be billed as four requests.





=========================================


SQS Developer Exam Tips


SQS - Delivery


  SQS Messages can be delivered multiple times and in any order.



SQS - Default Visibility Time Out


  Default Visibility Time Out is 30 seconds


  Maximum Time Out is 12 Hours



When you receive a message from a queue and begin processing it, you may find the visibility timeout for the queue is insufficient to fully process and delete that message. To give yourself more time to process the message, you can extend its visibility timeout by using the ChangeMessageVisibility action to specify a new timeout value. Amazon SQS restarts the timeout period using the new value.





SQS Long Polling


SQS long polling is a way to retrieve messages from your SQS queues. While the traditional SQS short polling returns immediately, even if the queue being polled is empty, SQS long polling doesn't return a response until a message arrives in the queue, or the long poll times out. SQS long polling makes it easy and inexpensive to retrieve messages from your SQS queue as soon as they are available.


Maximum Long Poll Time Out = 20 seconds





Example Questions


Polling in a tight loops is burning CPU cycles and costing the company money. How would you fix this? - To enable the long polling



SQS - Fanning Out


Create an SNS topic first using SNS. Then create and subscribe multiple SQS queues to the SNS topic.


Now whenever a message is sent to the SNS topic, the message will be fanned out to the SQS queues, i.e. SNS will deliver the message to all the SQS queues that are subscribed to the topic.




==========================




SQS Quiz


- SQS was the first service on the AWS platform? - true

- How large can an SQS message be? - 256kb

- What is the default visibility time out setting? - 30 seconds

- An SQS message can be delivered multiple times - True

- You are designing a new application which involves processing payments and delivering promotional emails to customers. You plan to use SQS to help facilitate this. You need to ensure that the payment process takes priority over the creation and delivery of emails. What is the best way to achieve this.

  : Use 2 SQS queues for the platform. Have the EC2 fleet poll the payment SQS queue first. If this queue is empty, then poll the promotional emails queue.

- Your EC2 instance download jobs from the SQS queue, however they are taking too long to process them. What API call can you use to extend the length of time to process the jobs? : ChangeMessageVisibility

- What is the default visibility time out? : 30 seconds

- You have a fleet of EC2 instances that are constantly polling empty SQS queues which is burning CPU compute cycles and costing your company money. What should you do?

  : Enable SQS Long Polling

- What is the maximum long poll time out : 20 seconds

- What amazon service can you use in conjunction with SQS to 'fan out' SQS messages to multiple queues : SNS



========================================





반응형


반응형

DynamoDB Summary



What is DynamoDB?


Amazon DynamoDB is a fast and flexible NoSQL database service for all applications that need consistent, single-digit millisecond latency at any scale. It is a fully managed database and supports both document and key-value data models. Its flexible data model and reliable performance make it a great fit for mobile, web, gaming, ad-tech, IoT, and many other applications.



Quick facts about DynamoDB


- Stored on SSD storage

- Spread Across 3 geographically distinct data centers


- Eventual Consistent Reads (Default)

  : Consistency across all copies of data is usually reached within a second. Repeating a read after a short time should return the updated data. (Best Read Performance)

  

- Strongly Consistent Reads

  : A strongly consistent read returns a result that reflects all writes that received a successful response prior to the read

  

  

The Basics


- Tables

- Items (Think a row of data in table)

- Attributes (Think of a column of data in a table)



DynamoDB - Primary Keys





Two Types of Primary Keys Avaliable


- Single Attribute (think unique ID)

  : Partition Key (Hash Key) composed of one attribute

  

- Composite (think unique ID and a date range)

  : Partition Key & Sort Key (Hash & Range) composed of two attributes.



  

- Partition Key

  : DynamoDB uses the partition key's value as input to an internal hash function. The output from the hash function determines the partition (this is simply the physical location in which the data is stored).

  : No two items in a table can have the same partition key value!

  

- Partition Key and Sort key

  : DynamoDB uses partition key's value as input to an internal hash function. The output from the hash function determines the partition (this is simply the physical location in which the data is stored)

  : Two items can have the same partition key, but they must have different sort key.

  : All items with the same partition key are stored together, in sorted order by sort key value.

  


DynamoDB - Indexes


- Local Secondary Index

  : Has the SAME partition key, different sort key.

  : Can ONLY be created when creating a table. They cannot be removed or modified later.



  

- Global Secondary Index

  : Has DIFFERENT Partition key and different sort key

  : Can be created at table creation or added LATER



  

  

DynamoDB - Streams


- Used to capture any kind of modification of the DynamoDB tables.

  : If a new item is added to the table, the stream captures an image of the entire item, including all of its attributes

  : If an item is updated, the stream captures the "before" and "after" image of any attributes that were modified in the item

  : If an item is deleted from the table, the stream captures an image of the entire item before it was deleted



  

  

Query & Scans Exam Tips


- A Query operation finds items in a table using only primary key attribute values. You must provide a partition key attribute name and a distinct value to search for.

- A Scan operation examines every item in the table. By default, a Scan returns all of the data attributes for every item, however, you can use the ProjectionExpression parameter so that the Scan only returns some of the attributes, rather than all of them.

- Try to use a query operation over a Scan operation as it is more efficient







Example 1


You have an application that requires to read 5 items of 10 KB per second using eventual consistency. What should you wet the read throughput to?


- First we  calculate how many read units per item we need

- 10 KB rounded up to nearest increment of 4 KB is 12 KB

- 12 KB / 4 KB = 3 read units per item


- 3 X 5 read items = 15

- Using eventual consistency we get 15/2 = 7.5


- 8 units of read throughput



Example 2 - Write THroughput


You have an application that requires to write 12 items of 100 KB per item each second. What should you set the write throughput to?


- Each write unit consist of 1 KB of data. You need to write 12 items per second with each item having 100 KB of data

- 12 X 100 KB = 1200 write units

- Write throughput of 1200 units



Erro Codes


400 HTTP Status Code - 

ProvisionedThroughputExceededException


You exceeded your maximum allowed provisioned throughput for a table or for one or more global secondary indexes.


 

Steps taken to authenticate


1. User Authenticates with ID provider (such as Facebook)

2. They are passed a Token by their ID provider

3. Your code calls AssumeRoleWithWebIdentity API and provides the providers token and specifies the ARN for the IAM Role

4. App can now access Dynamodb from between 15 minutes to 1 hour (default is 1 hour)



Conditional Writes.


If item = $10 then update to $12


Note that conditional writes are idempotent. This means that you can send the same conditional write request multiple times, but it will have no further effect on the item after the first time DynamoDB performs the specified update. For example, suppose you issue a request to update the price of a book item by 10%, with the expectation that the price is currently $20.

However, before you get a response, a network error occurs and you don't know whether your request was successful or not. Because a conditional update is an idempotent operation, you can send the same request again. and DynamoDB will update the price only if the current price is still $20.



Atomic Counters


DynamoDB supports atomic counters, where you use the UpdateItem operation to increment or decrement the value of an existing attribute without interfering with other write requests. (All write requests are applied in the order in which they were received.) For example, a web application might want to maintain a counter per visitor to their site. In this case, the application would need to increment this counter regardless of its current value.


Atomic Counter updates are not idempotent. This means that counter will increment each time you call UpdateItem. If you suspect that a previous request was unsuccessful, your application could retry the updateItem operation, however, this would risk updating the counter twice. This might be acceptable for a web site counter, because you can tolerate with slightly over- or under- counting the visitors. However, in a banking application, it would be safer to use a conditional update rather than an atomic counter.



Batch Operations


If your application needs to read multiple items, you can use the BatchGetItem API. A single BatchGetItem request can retrieve up to 1 MB of data, which can contain as many as 100 items, In addition, a single BatchGetItem request can retrieve items from multiple tables.



****** READ THE DYNAMODB FAQ ******


If you read one FAQ in preparing for this course, make sure it's the DynamoDB FAQ!!!!!





======================================


DynamoDB Quiz


- DynamoDB is a No-SQL database provided by AWS. - True

- You have a motion sensor which writes 600 items of data every minute. Each item consists of 5kb. Your application uses eventually consistent reads. What should you set the read throughput to ?

  : 600/60 = 10 items per second. 5kb rounded up = 8kb 8/4 = 2. 

  : 2 read per item. 2 X 10 = 20 reads per second. 

  : As the reads are Eventually consistent, 20/2 = 10

  ==> The answer is 10

- A scan is more efficient than a query in terms of performance - False

- What does the error "ProvisionedThroughputExceededException" mean in DynamoDB?

  : You exceeded your maximum allowed provisioned throughput for a table or for one or more global secondary indexes.

- You have a motion sensor which writes 600 items of data every minute. Each item consists of 5 kb. what should you set the write throughput to? : 50

- What is the API call to retrieve multiple items from a DynamoDB table?

  : BatchGetItem

- You have a motion sensor which writes 600 items of data every minute. Each item consists of 5kb. Your application uses strongly consistent read. What should you set read throughput to? 

  : 600/60 = 10 items per second

  : 5kb rounded to nearest 4 kb chunk is 8kb. 8/4 = 2 reads per item

  : 2 X 10 reads per second.

  ==> The answer is 20

- Using the AWS portal, you are trying to Scale DynamoDB past its preconfigured maximums. Which service can you increase by raising a ticket to AWS support?

  : http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html

  ==> Provisioned throughput limits

 - You have an application that needs to read 25 items of 13 kb in size per second. Your application uses eventually consistent reads. What should you set the read throughput to?

   : 13 kb - roundup to the nearrest 4kb = 16 kb. 16/4 = 4 reads per item

   : 25 items X 4 = 100

   : 100 / 2 = 50 (eventually consistent reads)

   ==> The answer is 50

- You have an application that needs to read 25 items of 13 kb in size per second. Your application uses strongly consistent reads. What should you set the read throughput to?

   : 13 kb - roundup to the nearrest 4kb = 16 kb. 16/4 = 4 reads per item

   : 25 items X 4 = 100 (Strongly consistent reads)

   ==> The answer is 100

   


=======================================




반응형

[AWS Certificate] Developer - DynamoDB memo

2017. 11. 14. 09:57 | Posted by 솔웅


반응형

DynamoDB from CloudGuru lectures



=====================================================

============= DynamoDB ====================

=====================================================



What is DynamoDB? (***********)






Amazon DynamoDB is a fast and flexible NoSQL database service for all applications that need consistent, single-digit millisecond latency at any scale. It is a fully managed database and supports both document and key-value data models. Its flexible data model and reliable performance make it a great fit for mobile, web, gaming, ad-tech, IoT, and many other applications.







Quick facts about DynamoDB


- Stored on SSD storage

- Spread across 3 geographically distinct data centers


- Eventual Consistent Reads (Default)

  : Consistency across all copies of data is usually reached within a second. Repeating a read after a short time should return the updated data. (Best Read Performance)

  

- Strongly Consistent Reads

  : A strongly consistent read returns a result that reflects all writes that received a successful response prior to the read.

  


The Basics


- Tables

- Items (Think a row of data in table)

- Attributes (Think of a column of data in a table)



Pricing


- Provisioned THroughput Capacity

  : Write Throughput $0.0065 per hour for every 10 units

  : Read Throughput $0.0065 per hour for every 50 units

  

- First 25 GB stored per month is free

- Storage costs of $0.25 GB per month there after.


Pricing Example


Let's assume that your application needs to perform 1 million writes and 1 million reads per day, while storing 28 GB of data.


First, you need to calculate how many writes and reads per seconds you need. 1 million evenly spread writes per day is equivalent to 1,000,000 (writes) / 24 (hours) / 60 (minutes) / 60 (seconds) = 11.6 writes per second.


A dynamoDB Write capacity unit can handle 1 write per second, so you need 12 write capacity units. For write throughput, you are charged on $0.0065 for every 10 units.


So ($0.0065/10) * 12 * 24 = $0.1872 per day.


Similarly, to handle 1 million strongly consistent reads per day, you need 12 read capacity units. For read throughput you are charged $0.0065 for every 50 units.


So ($0.0065/50) * 12 * 24 = $0.0374 per day.


Storage costs is $0.25 per GB per month. Lets assume our database is 28 GB. We get the first 25 GB for free so we only pay for 3 GB of storage which is $0.75 per month.


Total Cost = $0.1872 per day + $0.0374 per day Plus Storage of 0.75 per month


(30 X ($0.1872 + $0.0372)) $0.75 = $7.488


With free tier you get

25 read capacity units

25 write capacity units


Easiest way to learn DynamoDB?


- Let's start our first Lab


======================================================


Creating a DynamoDB Table


Create a Role - Dynamo full access

Create a instance - Assign the Role to the instance


#!/bin/bash

yum update -y

yum install httpd24 php56 git -y

service httpd start

chkconfig httpd on

cd /var/www/html

echo "<?php phpinfo();?>" > test.php

git clone https://github.com/acloudguru/dynamodb



1178578-C02NW6G1G3QD:AWS_SSH changsoopark$ ssh ec2-user@52.91.230.105 -i EC2KeyPair.pem.txt 

The authenticity of host '52.91.230.105 (52.91.230.105)' can't be established.

ECDSA key fingerprint is SHA256:Zo4LcW4QASmSaf4H4kg5ioPGeqLicxV8TsJ+/JTQVj0.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '52.91.230.105' (ECDSA) to the list of known hosts.


       __|  __|_  )

       _|  (     /   Amazon Linux AMI

      ___|\___|___|


https://aws.amazon.com/amazon-linux-ami/2017.09-release-notes/

[ec2-user@ip-172-31-85-82 ~]$ sudo su

[root@ip-172-31-85-82 ec2-user]# cd /var/www/html

[root@ip-172-31-85-82 html]# ls

dynamodb  test.php

[root@ip-172-31-85-82 html]# curl -sS https://getcomposer.org/installer | php

All settings correct for using Composer

Downloading...


Composer (version 1.5.2) successfully installed to: /var/www/html/composer.phar

Use it: php composer.phar


[root@ip-172-31-85-82 html]# php composer.phar require aws/aws-sdk-php

Do not run Composer as root/super user! See https://getcomposer.org/root for details

Using version ^3.38 for aws/aws-sdk-php

./composer.json has been created

Loading composer repositories with package information

Updating dependencies (including require-dev)

Package operations: 6 installs, 0 updates, 0 removals

  - Installing mtdowling/jmespath.php (2.4.0): Downloading (100%)         

  - Installing psr/http-message (1.0.1): Downloading (100%)         

  - Installing guzzlehttp/psr7 (1.4.2): Downloading (100%)         

  - Installing guzzlehttp/promises (v1.3.1): Downloading (100%)         

  - Installing guzzlehttp/guzzle (6.3.0): Downloading (100%)         

  - Installing aws/aws-sdk-php (3.38.0): Downloading (100%)         

guzzlehttp/guzzle suggests installing psr/log (Required for using the Log middleware)

aws/aws-sdk-php suggests installing aws/aws-php-sns-message-validator (To validate incoming SNS notifications)

aws/aws-sdk-php suggests installing doctrine/cache (To use the DoctrineCacheAdapter)

Writing lock file

Generating autoload files

[root@ip-172-31-85-82 html]# cd dynamodb

[root@ip-172-31-85-82 dynamodb]# ls -l

total 24

-rw-r--r-- 1 root root  4933 Nov  9 00:32 createtables.php

-rw-r--r-- 1 root root    11 Nov  9 00:32 README.md

-rw-r--r-- 1 root root 11472 Nov  9 00:32 uploaddata.php

[root@ip-172-31-85-82 dynamodb]# nano createtables.php

==> update the Region info - create and update php




http://52.91.230.105/dynamodb/createtables.php


==> will create 4 dynamoDB tables


==> 

Creating table ProductCatalog... Creating table Forum... Creating table Thread... Creating table Reply... Waiting for table ProductCatalog to be created. Table ProductCatalog has been created. Waiting for table Forum to be created. Table Forum has been created. Waiting for table Thread to be created. Table Thread has been created. Waiting for table Reply to be created. Table Reply has been created.


Picture : DynamoDBCreated


http://52.91.230.105/dynamodb/uploaddata.php





===============================================


DynamoDB Indexes & Streams


* Primary Keys


Tow Types of Primary Keys available

- Single Attribute (think unique ID)

  : Partition Key (Hash Key) composed of one attribute


- Composite (think unique ID and a date range)

  : Partition Key & Sort Key (Hash & Range) composed of two attributes

  

Partition Key

- DynamoDB uses the partition key's value as input to an internal hash function. The output from the hash function determines the partition (this is simply the physical location in which the data is stored)

- No two items in a table can have the same partition key value (*****)


Partition Key and Sort key

- DynamoDB uses the partition key's value as input to an internal hash function. the output from the hash function determines the partition (this is simply the physical location in which the data is stored)

- Two items can have the same partition key, but they must have a different sort key

- All items with the same partition key are stored together, in sorted order by sort key value


* Indexes (***)


Local Secondary Index

- Has the SAME Partition key, different sort key

- Can ONLY be created when creating a table. They cannot be removed or modified later.


Global Secondary Index

- Has DIFFERENT Partition key and different sort key

- Can be created at table creation or added LATER


Used to capture any kind of modification of the DynamoDB tables

- If a new item is added to the table, the stream captures an image of the entire item, including all of its attributes

- If an item is updated, the stream captures the "before" and "after" image of any attributes that were modified in the item

- If an item is deleted from the table, the stream captures an image of the entire item before it was deleted


DynamoDB Streams






Practice - Tabs

Overview, Items, Metrics, Alarms, Capacity, Indexes, Triggers, Access control, Tags


=========================================


Scan vs. Query API Calls



What is a Query?


- A Query operation finds items in a table using only primary key attribute values. You must provide a partition attribute name and a distinct value to search for.


- You can optionally provide a sort key attribute name and value, and use a comparison operator to refine the search results.


- By default, a Query returns all of the data attributes for items with the specified primary key(s); however, you can use the ProjectionExpression parameter so that the Query only returns some of the attributes, rather than all of them


- Query results are always sorted by the sort key. If the data type of the sort key is a number, the results are returned in numeric order. otherwise, the results are returned in order of ASCII character code values. By default, the sort order is ascending. To reverse the order set the ScanIndexForward parameter to false.


- By Default is eventually consistent but can be changed to be strongly consistent.



What is a Scan?


- A Scan operation examines every item in the table. By default, a Scan returns all of the data attributes for every item. however, you can use the ProjectionExpression parameter so that the Scan only returns some of the attributes, rather than all of them


What should I use? Query vs. Scan?


Generally, a Query operation is more efficient than a Scan operation.


A Scan operation always scans the entire table, then filters out values to provide the desired result, essentially adding the extra step of removing data from the result set. Avoid using a Scan operation on a large table with a filter that removes many results, if possible. Also, as a table grows, the Scan operation slows. The Scan operation examines every item for the requested values, and can use up the provisioned throughput for a large table in a single operation


For quicker response times, design your tables in a way that can use the Query, Get, or BatchGetItem APIs, instead. Alternatively, design your application to use Scan operations in a way that minimizes the impact on your table's request rate.





Query & Scans Exam Tips


- A Query operation finds items in a table using only primary key attribute values. You must provide a partition key attribute name and a distinct value to search for


- A Scan operation examines every item in the table. By default, a Scan returns all of the data attributes for every item. however, you can use the ProjectionExpression parameter so that the Scan only returns some of the attributes, rather than all of them


- Query results are always sorted by the sort key in ascending order. Set ScanIndexForward parameter to false to reverse it.


- Try to use a query operation over a Scan operation as it is more efficient


=======================================


DynamoDB Provisioned Throughput Calculations (***)


- Unit of Read provisioned throughput

  : All reads are rounded up to increments of 4KB

  : Eventually Consistent Reads (default) consist of 2 reads per second

  : Strongly Consistent Reads consist of 1 read per second

  

- Unit of Write provisioned throughput

  : All writes are 1 KB

  : All writes consist of 1 write per second

  

The Magic Formula


Question 1 - You have an application that requires to read 10 items of 1 KB per second using evnetual consistency. What should you set the read throughput to?


(Size of Read rounded to nearest 4 KB chunk/ 4KB) X no of items = read throughput


Divide by 2 if eventually consistent


- First we calculate how many read units per item we need


- 1 KB rounded to the nearest 4 KB increment = 4

- 4 KB / 4KB = 1 read unit per item


- 1 X 10 read items = 10

- Using eventual consistency we get 10 / 2 = 5

- 5 units of read throughput



Question 2

You have an application that requires to read 10 items of 6 KB per second using eventual consistency. What should you set the read throughput to?


- First we calculate how many read units per item we need

- 6 KB rounded up to nearest increment of 4 KB is 8 KB

- 8 KB / 4 KB = 2 read units per item


- 2 X 10 read items = 20

- Using eventual consistency we get 20 / 2 = 10


- 10 units of read throughput



Question 3


You have an application that requires to read 5 items of 10 KB per second using eventual consistency. What should you set the read throughput to?


- First we calculate how many read units per item we need

- 10 KB rounded up to nearest increment of 4 KB is 12 KB

- 12 KB / 4 KB = 3 read units per item.


- 3 X 5 read items = 15

- Using eventual consistency we get 15 / 2 = 7.5


- 8 units of read throughput



Question 4 - STRONG CONSISTENCY


You have an application that requires to read 5 items of 10 KB per second using strong consistency. What should you set the read throughput to?


- First we calculate how many read units per item we need 

- 10 KB rounded up to nearest increment of 4 KB is 12 KB

- 12 KB / 4 KB = 3 read units per item


- 3 X 5 read items = 15

- Using strong consistency we Don't divide by 2


- 15 units of read throughput



Question 5 - WRITE THROUGHPUT


You have an application that requires to write 5 items, with each item being 10 KB in size per second. What should you set the write throughput to?


- Each write unit consist of 1 KB of data. You need to write 5 items per second with each item using 10 KB of data


- 5 X 10 KB = 50 write units


- Write throughput of 50 Units



Question 6 - WRITE THROUGHPUT


You have an application that requires to write 12 items of 100 KB per item each second. What should you set the write throughput to?


- Each write unit consist of 1 KB of data. You need to write 12 items per second with each item having 100 KB of data.


- 12 X 100 KB = 12 write units


- Write throughput of 1200 Units



Error Code


400 HTTP Status Code - ProvisionedTHroughputExceededException


You exceeded your maximum allowed provisioned throughput for a table or for one or more global secondary indexes.





========================================





Using Web Identity Providers with DynamoDB


Web Identity Providers


You can authenticate users using Web Identity providers (such as Facebook, Google, Amazon or any other Open-ID Connect-compatible Identity provider). This is done using AssumeRoleWithWebIdentity API.


You will need to create a role first.


1. Web Identity Token

2. App ID of provider

3. ARN of Role

a. AccessKeyID

   SecretAccessKey

   SessionToken

b. Expiration (time limit)

c. AssumeRoleID

d. SubjectFromWebIdentityToken

(the unique ID that appears in an IAM policy variable for this particular identity provider)



Steps taken to authenticate


1. User Authenticates with ID provider (such as facebook)

2. They are passed a Token by their ID provider

3. Your code calls AssumeRoleWithWebIdentity API and provides the providers token and specifies the ARN for the IAM Role

4. App can now access Dynamodb from between 15 minutes to 1 hour (default is 1 hour)


========================================


Other important aspects of DynamoDB


Conditional Writes




If item = $10 then update to $12


Note that conditional writes are idempotent. This means that you can send the same conditional write request multiple times, but it will have no further effect on the item after the first time DynamoDB performs the specified update. For example, suppose you issue a request to update the price of a book item by 10%, with the expectation that the price is currently $20. However, before you get a response, a network error occurs and you don't know whether your request was successful or not. Because a conditional update is an idempotent operation, you can send the same request again. and DynamoDB will update the price only if the current price is still $20.



Atomic Counters


DynamoDB supports atomic counters, where you use UpdateItem operation to increment or decrement the value of an existing attribute without interfering with other write requests. (All write requests are applied in the order in which they were received.) For example, a web application might want to maintain a counter per visitor to their site. In this case, the application would need to increment this counter regardless of its current value.



Batch Operations


If your application needs to read multiple items, you can use the BatchGetItem API. A single BatchGetItem request can retrieve up to 1 MB of data, which can contain as many as 100 items. In addition, a single BatchGetItem request can retrieve items from multiple tables.



===============================================




반응형


반응형

CloudGuru (Udemy lecture)


AWS Certified Developer - Associate 2017



================================================================

============= Databases Overview & Concepts ====================

================================================================


Database 101



This section is not so much in the Exam. This is just for fundamental knowledge on Database.

(DynamoDB is mostly in the Exam and we will learn it from next article)



What is Relational database?



Relational databases are what most of us are all used to. They have been around since the 70's. Think of a traditional spreadsheet

- Database

- Table

- Row

- Fields (Columns)


Relational Database Types

- SQL Server

- Oracle

- MySQL Server

- PostgreSQL

- Aurora

- MariaDB



Non Relational Databases


- Database

  : Collection ==> Table

  : Document ==> Row

  : Key Value Pairs ==> Fields



JSON/NoSQL


Sample






What is Data Warehousing?


Used for business intelligence. Tools like cognos, jaspersoft, SQL Server Reporting Services, Oracle Hyperion, SAP NetWeaver.


Used to pull in very large and complex data sets. Usually used by management to do queries on data (such as current performance vs. targets etc.)



OLTP vs. OLAP


Online Transaction Processing (OLTP) differs from OLAP Online Analytics Processing (OLAP) in terms of the types of queries run.




OLTP Example:


Order number 2120121

Pulls up a row of data such as Name, Date, Address to Deliver to , Delivery Status etc.


OLAP


OLAP transaction Example:

Net Profit for EMEA and pacific for the Digital Radio Product.

Pulls in large numbers of records


Sum of Radios Sold in EMEA

Sum of Radios Sold in Pacific

Unit Cost of Radio in each region

Sales price of each radio

Sales price - unit cost.


Data Warehousing databases use different type of architecture both from a database perspective and infrastructure layer.



What is Elasticache?




ElastiCache is a web service that makes it easy to deploy, operate, and scale an in-memory cache in the cloud. The service improves the performance of web applications by allowing you to retrieve information from fast, managed, in-memory caches, instead of relying entirely on slower disk-based databases. ElasticCache supports two open-source in-memory caching engines:

- Memcached

- Redis



What is DMS?




Announced at re:Invent 2015, DMS stands for Database Migration Service.

Allows you to migrate your production database to AWS. Once the migration has started, AWS mansges all the complexities of the migration process like data type transformation, compression, and parallel transfer (for faster data transfer) while ensuring that data changes to the source database that occur during the migration process are automatically replicated to the target.


AWS schema conversion tool automatically converts the source database schema and a majority of the custom code, including views, stored procedures, and functions, to a format compatible with the target database.





AWS Database Types - Summary


RDS - OLTP

  : SQL

  : MySQL

  : PostgreSQL

  : Oracle

  : Aurora

  : MariaDB


DynamoDB - No SQL

Redshift - OLAP

Elasticache - In Memory Caching

  : Memcached

  : Redis

DMS





반응형


반응형

S3 Summary



* Remember that S3 is Object based i.e. allows you to upload files.

* Files can be from 0 Bytes to 5TB

* There is unlimited storage

* Files are stored Buckets

* S3 is a universal namespace, that is, names must be unique globally

* name - i.e. https://s3-eu-west-1.amazonaws.com/acloudgutu


* Read after Write consistency for PUTs of new Objects

* Eventual Consistency for overwrite PUTS and DELETES (can take some time to propagate)

* S3 Storage Classes/Tiers

  : S3 (durable, immediately available, frequently accessed)

  : S3 - IA (durable, immediately available, infrequently accessed)

  : Reduced Redundancy Storage (data that is easily reproducible, such as thumb nails etc).

  : Glacier - Archived data, where you can wait 3 - 5 hours before accessing


* Remember the core fundamentals of S3

  : Key (name)

  : Value (data)

  : Version ID

  : Metadata

  : Access Control lists

  

* Object based storage only (for files)

* Not suitable to install an operating system on (***)

 


Versioning





* Stores all versions of an object (including all writes and even if you delete an object)

* Great backup tool

* Once enabled, Versioning cannot be disabled, only suspended.

* Integrates with Lifecycle rules

* Versioning's MFA Delete capability, which uses multi-factor authentication, can be used to provide an additional layer of security

* Cross Region Replication, requires versioning enabled on the source bucket

  


Lifecycle Management





* Can be used in conjunction with versioning

* Can be applied to current versions and previous versions

* Following actions can now be done

  : Transition to the Standard-Infrequent Access Storage Class (128Kb and 30 days after the creation date)

  : Archive to the Glacier Storage Class (30 days after IA, if relevant)

  : Permanently Delete

  


CloudFront



* Edge Location - This is the location where content will be cached. This is separate to an AWS Region/AZ

* Origin - This is the origin of all the files that the CDN will distribute. This can be either an S3 Bucket, an EC2 Instance, an Elastic Load Balancer or Route53

* Distribution - This is the name given the CDN which consists of a collection of Edge Locations.

  : Web Distribution - Typically used for Websites

  : RTMP - Used for Media Streaming

* Edge locations are not just READ only, you can write to them too. (i.e. put an object on to them)

* Objects are cached for the life of the TTL (Time To Live)

* You can clear cached objects, but you will be charged.



Securing your buckets



* By default, all newly created buckets are PRIVATE

* You can setup access control to your buckets using

  : Bucket Policies

  : Access Control Lists

* S3 buckets can be configured to create access logs which log all requests made to the S3 bucket. This can be done to another bucket.



Encryption



* In Transit

  : SSL/TLS

* At Rest

  : Server Side Encryption

    - S3 Managed Keys - SSE-S3 (***)

    - AWS Key Management Service, Managed Keys - SSE-KMS (***)

    - Server Side Encryption With Customer Provided Keys - SSE-C (***)

* Client Side Encryption



Storage Gateway



* File Gateway - For flat files, stored directly on S3

* Volume Gateway

  : Stored Volumes - Entire Dataset is stored on site and is asynchronously backed up to S3.

  : Cached Volumes - Entire Dataset is stored on S3 and the most frequently accessed data is cached on site

* Gateway Virtual Tape Library (VTL)

  : Used for backup and uses popular backup applications like NetBackup, Backup Exec, Veam etc.






Snowball



* Snowball

* Snowball Edge

* Snowmobile


* Understand what Snowball is

* Understand what Import Export is

* Snowball Can

  : Import to S3

  : Export from S3

  


S3 Transfer Acceleration



* You can speed up transfers to S3 using S3 transfer acceleration. This costs extra, and has the greatest impact on people who are in far away location.



S3 Static Websites



* You can use S3 to host static websites

* Serverless

* Very cheap, scales automatically

* STATIC only, cannot host dynamic sites



CORS



* Cross Origin Resource Sharing

* Need to enable it on the resources bucket and state the URL for the origin that will be calling the bucket.

i.e. 

http://mybucketname.s3-website.eu-west-2.amazonaws.com - S3 Website

https://s3.eu-west-2.amazonaws/mybucketname      - Bucket



Last few tips



* Write to S3 - HTTP 200 code for a successful write

* You can load files to S3 much faster by enabling multipart upload

* Read the S3 FAQ before taking the exam. It comes up A LOT!





=====================================



S3 Quiz



* The minimum file size allowed on S3 is 0 bytes? True

* If you encrypt a bucket on S3 what encryption does AWS use? 

  ==> Advanced Encryption Standard (AES) 256

* You create a static hosting website in a bucket called "acloudguru" in Japan using S3. What would the new URL End Point be? 

  ==> http://acloudguru.s3-website-ap-northeast-1.amazonaws.com

* You are hosting a static website in an S3 bucket which uses Java script to reference assets in another S3 bucket. For some reason however these assets are not displaying when users browse to the site. What could be the problem?

  ==> You haven't enabled Cross Origin Resource Sharing (CORS) on the bucket where the assets are stored

* What is the HTTP code you would see if once you successfully place a file in an S3 bucket? ==> 200


* S3 provides unlimited storage. ==> True

* What is the maximum file size that can be stored on S3? ==> 5Tb

* What is the largest size file you can transfer to S3 using a PUT operation? 

  ==> The correct answer is 5Gb. After that you must use a multipart upload. This can be an exam question. Please remember this before going in to your exam. Correct! http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html

* If you want to enable a user to download your private data directly from S3, you can insert a pre-signed URL into a web page before giving it to your user. ==> True

* When you first create an S3 bucket, this bucket is publicly accessible by default. ==> False





반응형


반응형

S3 ( Simple Storage Service)


S3 provides developers and IT teams with secure, durable, highly-scalable object storage. Amazon S3 is easy to use, with a simple web services interface to store and retrieve any amount of data from anywhere on the web.


S3 is a safe place to store your files.

It is Object based storage.

The data is spread across multiple devices and facilities.


The Basics

- S3 is Object based i.e. allows you to upload files.

- Files can be from 0 Bytes to 5TB

- There is unlimited storage

- Files are stored in Buckets.

- S3 is a universal namespace, that is, names must be unique globally.

- https://s3-eu-west-1.amazonaws.com/acloudguru

- When you upload a file to S3 you will receive a HTTP 200 code if the upload was successful.


Data Consistency Model For S3 (***)

- Read after Write consistency for PUTS of new objects

- Eventual Consistency for overwrite PUTS and DELETES (can take some time to propagate)


S3 is a simple key, value store

- S3 is Object based. Objects consist of the following

: Key (this is simply the name of the object)

: Value (This is simply the data and is made up of a sequence of bytes)

: Version ID (Important for versioning)

: Metadata (Data about the data you are storing)

: Subresources

  Access Control Lists

  Torrent

: Built for 99.99% availability for the S3 platform

: Amazon guarantees 99.999999999% durability for S3 information (Remember 11X9's)

: Tiered Storage Available

: Lifecycle Management

: Versioning

: Encryption

: Secure your data using Access Control Lists and Bucket Policies


Storage Tiers/Classes

: S3 - 99.99% availability, 99.999999999% durability, stored redundantly across multiple devices in multiple facilities and is designed to sustain the loss of 2 facilities concurrently

: S3 - IA (Infrequently Accessed) For data that is accessed less frequently, but requires rapid access when needed. Lower fee than S3, but you are charged a retrieval fee.

: Reduced Redundancy Storage - Designed to provide 99.99% durability and 99.99% availability of objects over a given year.

: Glacier - Very cheap, but used for archival only. It takes 3-5 hours to restore from Glacier



What is Glacier?


Glacier is an extremely low-cost storage service for data archival. Amazon Glacier stores data for as little as $0.01 per gigabyte per month, and is optimized for data that is infrequently accessed and for which retrieval times of 3 to 5 hours are suitable.



S3- Charges

-  Charged for

: Storage

: Requests

: Storage Management Pricing

: Data Transfer Pricing

: Transfer Acceleration



What is S3 Transfer Acceleration?



Amazon S3 Transfer Acceleration enables fast, easy and secure transfers of files over long distances between your end users and an S3 bucket.

Transfer Acceleration takes advantage of Amazon CloudFront's globally distributed edge location. As the data arrives at an edge location, data is routed to Amazon S3 over an optimized network path.


Exam Tips for S3 101

- Remember that S3 is Object based i.e. allows you to upload files.

- Files can be from 0 Bytes to 5TB

- There is unlimited storage

- Files are stored in Buckets

- S3 is a universal namespace, that is, names must be unique globally.

- https://s3-eu-west-1.amazonaws.com/acloudguru

- Read after Write consistency for PUTS of new Objects

- Eventual Consistency for overwrite PUTS and DELETES (can take some time to propagate)

- S3 Storage Classes/Tiers

: S3 (durable, immediately available, frequently accessed)

: S3 - IA (durable, immediately available, infrequently accessed)

: S3 - Reduced Redundancy Storage (data that is easily reproducible, such as thumb nails etc)

: Glacier - Archived data, where you can wait 3-5 hours before accessing.

- Remember the core fundamentals of an S3 object

: key (name)

: value (data

: Version ID

: Metadata

: Subresources

  ACL

  Torrent

- object based storage only (for files) (*****)

- Not suitable to install an operating system on. (*****)

- Successful uploads will generate a HTTP 200 status code.


- Read the S3 FAQ before taking the exam It comes up A LOT! (*****)


================


S3 Essencial - 


Bucket is just folder where you can upload files


- Buckets are a universal name space

- Upload an object to S3 receive a HTTP 200 Code

- S3, S3 - IA, S3 Reduced Redundancy Storage

- Encryption

: Client Side Encryption

: Server Side Encryption

  Server side encryption with Amazon S3 Managed Keys (SSE-S3)

  Server side encryption with KMS (SSE-KMS)

  Server side encryption with Customer Provided Keys (SSE-C)

- Control access to buckets using either a bucket ACL or using Bucket Polices

- BY DEFAULT BUCKETS ARE PRIVATE AND ALL OBJECTS STORED INSIDE THEM ARE PRIVATE


===================


Create a S3 Website


Static page only, no dynamic page (PHP etc.)

Format of URL : bucketname.s3-website-region.amazonaws.com


===================


Cross Origin Resource Sharing (CORS)


Cross-origin resource sharing (CORS) defines a way for client web applications that are loaded in one domain to interact with resources in a different domain. With CORS support in Amazon S3, you can build rich client-side web applications with Amazon S3 and selectively allow cross-origin access to your Amazon S3 resources.



Lambda -> Create Functions

Triggers of Lambda function - ?? *****

Amazon API Gateway

upload html files to S3

IAM - Role, Policy

Route 53 
- Register Domain

Is it really serverless?
- Vendor takes care of provisioning and management of servers
- Vendor is responsible for capacity provisioning and automated scaling
- Moving away from servers and infrastructure concerns should be your goal

=====================







Using Polly to help you pass your exam - A serverless approach

Polly 
- Text-to-Speech : Type statements -> can download it to mp3

Create S3 bucket - 2 buckets

Simple Notification Service (SNS)

DynamoDB table

IAM - create new role : Lambda - Add permissions - attach new policy

Lambda - Create 2 Lambda functions

Add Trigger : SNS 


============================

Using Polly to help you pass your exam - A serverless approach : Part 2

Create 3rd Lambda function (PostReader_GetPosts)

Amazon API Gateway - Create new API (PostReaderAPI)

Go to S3 and deploy the website

=============================


=============================

S3 - Versioning

S3 - Create a Bucket - Enable versioning

Bucket - upload a text file to the bucket - update the file and upload it again
- Click on Latest Version link -> can select a version from dropdown list

Delete the text file - initiate restore => can restore the deleted file
Actions - Delete the Delete Marker

* Stores all version sof an object (including all writes and even if you delete an object)
* Great backup tool
* Once enabled, Versioning cannot be disabled, only suspended.
* Integrates with Lifecycle rules 
* Versioning's MFA Delete capability, which uses multi-factor authentication, can be used to provide an additional layer of security.

==============================================

Cross region replication

S3 - Create a new bucket
Existing and new bucket would be in different region

Existing bucket - Management - Replication - Add Rule - Select options - Select Destination (new bucket) - Enable versioning - Change the storage class - Select IAM role - Save 
Replication enabled
Go to new bucket - not replicated yet
Commend line - pip install awscli etc. 

IAM - Create Group - Attach Policy 
Create a User - Access key ID - Secret....
Terminal - aws configure
Access key ID - 
Secret Access Key - 
default region name - 

aws s3 ls - will show buckets (now there are 2 buckets)

aws s3 cp --recursive s3://existing_bucket s3://new_bucket -> will copy the contents from existing to new bucket

Back to console and check the new bucket - will be the objects from existing bucket

* Versioning must be enabled on both the source and destination buckets.
* Regions must be unique
* Files in an existing bucket are not replicated automatically. All subsequent updated files will be replicated automatically
* You cannot replicate to multiple buckets or use daisy chaining (at this time.)
* Delete markers are replicated
* Deleting individual versions or delete markers will not be replicated
* Understand what Cross Region Replication is at a high level

===================================


Glacier - Data Archival

S3 - Create a bucket - Enable Versioning - all default 

Management - Lifecycle - add lifecycle rule - rule name - Current version, select transition to standard-IA after 30 days - add transition - Select transition to Amazon Glacier after 60 days - previous version - Transition to Standard-IA after 30 days - Select transition to Amazon Glacier after 60 days - Configure expiration - Current/previous version expire after 425 days - Save

* Can be used in conjunction with versioning
* Can be applied to current versions and previous versions
* Following actions can now be done
  - Transition to the Standard - infrequent Access Storage Class (128kb and 30 days after the creation date)
  - Archive to the Glacier Storage Class (30 days after IA, if relevant)
  - Permanently Delete
  
============================================

Cloud Front Overview





A content delivery network (CDN) is a system of distributed servers (network) that deliver webpages and other web content to a user based on the geographic locations of the user, the origin of the webpage and a content delivery server.

CloudFront - Key Terminology
* Edge Location - THis is the location where content will be cached. This is separate to an AWS Region/AZ
* Origin - THis is the origin of all the files that the CDN will distribute. This can be either an S3 Bucket, an EC2 instance, an Elastic Load Balancer or Route 53
* Distribution - THis is the name given the CDN which consistes of a collection of Edge Locations

What is CloudFront

Amazon CloudFront can be used to deliver your entire website, including dynamic, static, streaming, and interactive content using a global network of edge locations. Requests for your content are automatically routed to the nearest edge location, so content is delivered with the best possible performance.

Amazon CloudFront is optimized to work with other Amazon Web Services, like Amazon Simple Storage Service (Amazon S3), Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Load Balancing, and Amazon Route 53. Amazon CloudFront also works seamlessly with any non-AWS origin server, which stores the original, definitive versions of your file.



CloudFront - Key Terminology

* Web Distribution - Typically used for Websites
* RTMP - Used for Media Streaming

CloudFront - Exam Tips

* Edge Location - This is the location where content will be cached. This is separate to an AWS Region/AZ
* Origin - This is the origin of all the files that the CDN will distribute. This can  be either an S3 Bucket, an EC2 Instance, an Elastic Load Balancer or Route53
* Distribution - This is the name given the CDN which consists of a collection of Edge Locations
  - Web Distribution - Typically used for Websites
  - RTMP - Used for Media Streaming
* Edge locations are not just READ only, you can write to them too. (i.e. put an object on to them)
* Objects are cached for the life of the TTL (Time To Live)
* You can clear cached objects, but you will be charged.

=======================================================

Create CDN

S3 - Create a bucket - upload a file - public permission 
Cloud Front - Service - Distribution - get started (web) - fill in fields - Create

Exam Topic - Distribution - Web, RTMP *****
- Restriction Type : Whitelist, Blacklist
- Invalidations : 

S3- goto Bucket - open the file uploaded  ==> Go to CloudFront - Copy domain name - enter the domain name + /uploaded file name ==> loading faster

CloudFront - Paid service

==========================================================



==========================================================

S3 - Security & Encryption

* By default, all newly created buckets are PRIVATE
* You can setup access control to your buckets using
  - Bucket Policies
  - Access Control Lists
* S3 buckets can be configured to create access logs which log all requests made to the S3 bucket. This can be done to another bucket.

Encryption
* In Transit : 
  - SSL/TLS
* At Rest
  - Server Side Encryption
    : S3 Managed Keys - SSE-S3
    :AWS Key Management Service, managed Keys - SSE-KMS
    : Server Side Encryption with Customer Provided Keys - SSE-C
  - Client Side Encryption
    
==============================================





AWS Storage Gateway is a service that connects an on-premises software appliance with cloud-based storage to provide seamless and secure integration between an organization's on-premises IT environment and AWS's storage infrastructure. The service enables you to securely store data to the AWS cloud for scalable and cost-effective storage.

AWS Storage Gateway's software appliance is available for download as a virtual machine (VM) image that you install on a host in your datacenter. Storage Gateway supports either VMware ESXi or Microsoft Hyper-V. Once you've installed your gateway and associated it with your AWS account through the activation process, you can use the AWS Management Console to create the storage gateway option that is right for you.

* Four Types of Storage Gateways
- File Gateway (NFS)
- Volumes Gateway (iSCSI)
  : Stored Volumes
  : Cache Volumes
- Tape Gateway (VTL)

* File Gateway
Files are stored as objects in your S3 buckets, accessed through a Network File System (NFS) mount point. Ownership, permissions, and timestamps are durably stored in S3 in the user-metadata of the object associated with the file. Once objects are transferred to S3, they can be managed as native S3 objects, and bucket policies such as versioning, lifecycle management, and cross-region replication apply directly to objects stored in your bucket.

* Volume Gateway
The volume interface presents your applications with disk volumes using the iSCSI block protocol.
Data written to these volumes can be asynchronously backed up as point-in-time snapshots of your volumes, and stored in the cloud as Amazon EBS snapshots.
Snapshots are incremental backups that capture only changed blocks. All snapshot storage is also compressed to minimize your storage charges.

* Stored Volumes
Stored volumes let you store your primary data locally, while asynchronously backing up that data to AWS. Stored volumes provide your on-premises applications with low-latency access to their entire datasets, while providing durable, off-site backups. You can create storage volumes and mount them as iSCSI devices from your on-premises application servers. Data written to your stored volumes is stored on your on-premises storage hardware. This data is asynchronously backed up to Amazon Simple Storage Service (Amazon S3) in the form of Amazon Elastic Block Store (Amazon EBS) snapshots. 1GB - 16 TB in size for stored Volumes.

* Cached Volumes
Cached volumes let you use Amazon Simple Storage Service (Amazon S3) as your primary data storage while retaining frequently accessed data locally in your storage gateway. Cached volumes minimize the need to scale your on-premises storage infrastructure, while still providing your applications with low-latency access to their frequently accessed data. You can create storage volumes up to 32 TiB in size and attach to them as iSCSI devices from your on-premises application servers. Your gateway stores data that you write to these volumes in Amazon S3 and retains recently read data in your on-premises storage gateway's cache and upload buffer storage. 1GB-32TB in size for Cached Volumes.

* Tape Gateway
Tape Gateway offers a durable, cost-effective solution to archive your data in the AWS Cloud. The VTL interface it provides lets you leverage your existing tape-based backup application infrastructure to store data on virtual tape cartridges that you create on your tape gateway. Each tape gateway is preconfigured with a media changer and tape drives, which are available to your existing client backup applications as iSCSI devices. You add tape cartridges as you need to archive your data. Supported by NetBackup, Backup Exec, Veam etc.

Exam Tips

- File Gateway - For flat files, stored directly on S3.
- Volume Gateway
  : Stored Volumes - Entire Dataset is stored on site and is asynchronously backed up to S3
  : Cached Volumes - Entire Dataset is stored on S3 and the most frequently accessed data is cached on site.
- Gateway Virtual Tape Library (VTL)
  : Used for backup and uses popular backup applications like NetBackup, Backup Exec, Veam etc.
  
=======================================


Import/Export Disk

AWS Import/Export Disk accelerates moving large amounts of data into and out of the AWS cloud using portable storage devices for transport. AWS Import/Export Disk transfers your data directly onto and off of storage devices using Amazon's high-speed internal network and bypassing the Internet.

Types of Snowballs
* Snowball
* Snowball Edge
* Snowmobile




* Snowball
Snowball is a petabyte-scale data transport solution that uses secure appliances to transfer large amounts of data into and out of AWS. Using Snowball addresses common challenges with large-scale data transfers including high network costs, long transfer times, and security concerns. Transferring data with Snowball is simple, fast, secure, and can be as little as one-fifth the cost of high-speed Internet.

80TB snowball in all regions. Snowball uses multiple layers of security designed to protect your data including tamper-resistant enclosures, 256-bit encryption, and an industry-standard Trusted Platform Module (TPM) designed to ensure both security and full chain-of-custody of your data. Once the data transfer job has been processed and verified, AWS performs a software erasure of the Snowball appliance.

* Snowball Edge
AWS Snowball Edge is a 100TB data transfer device with on-board storage and compute capabilities. You can use Snowball Edge to move large amounts of data into and out of AWS, as a temporary storage tier for large local datasets, or to support local workloads in remote or offline locations.

Snowball Edge connects to your existing applications and infrastructure using standard storage interfaces, streamlining the data transfer process and minimizing setup and integration. Snowball Edge can cluster together to form a local storage tier and process your data on-premises, helping ensure your applications continue to run even when they are not able to access the cloud.

* Snowmobile
AWS Snowmobile is an Exabyte-scale data transfer service used to move extremely large amounts of data to AWS. You can transfer up to 100PB per Snowmobile, a 45-foot long ruggedized shipping container, pulled by a semi-trailer truck. Snowmobile makes it easy to move massive volumes of data to the cloud, including video libraries, image repositories, or even a complete data center migration. Transferring data with Snowmobile is secure, fast and cost effective.

Exam Tips

* Understand what Snowball is
* Understand what Import Export is
* Snowball Can
  : Import to S3
  : Export from S3
  
==================================================


S3 Transfer Acceleration utilities the CloudFront Edge Network to accelerate your uploads to S3. Instead of uploading directly to your S3 bucket, you can use a distinct URL to upload directly to an edge location which will then transfer that file to S3. You will get a distinct URL to upload to acloudguru.s3-accelerate.amazonaws.com

S3 - Create a bucket 
Properties - Transfer acceleration - Enabled - Click on the link in the popup window - Check upload speed (Speed Comparison)

========================================





반응형


반응형


EC2 - Summary & Exam TIps


From Cloud Guru lecture in udemy






* Know the differences (pricing models) between (***)

- On Demand 

- Spot

- Reserved

- Dedicated Hosts : 


==> Choose best pricing model for specific requests


* Remember with spot instances;

- If you terminate the instance, you pay for the hour

- if AWS terminates the spot instance, you get the hour it was terminated in for free.



* EC2 Instance Types


Making Sense of AWS EC2 Instance Type Pricing: ECU Vs. vCPU





EBS (Elastic Block Store) Consists of;

- SSD, General Purpose - GP2 (Up to 10,000 IOPS)

- SSD, Provisioned IOPS - I01 (More than 10,000 IOPS)

- HDD, THroughput Optimized - ST1 - frequently accessed workloads

- HDD, Cold - SC1 - less frequently accessed data.

- HDD, Magnetic - Standard - cheap, infrequently accessed storage


* You cannot mount 1 EBS volume to multiple EC2 instances, instead use EFS.



EC2 Lab Exam Tips

* Termination Protection is turned off by default, you must turn it on

* On an EBS-backed instance, the default action is for the root EBS volume to be deleted when the instance is terminated

* Root volumnes cannot be encrypted by default, you need a third party tool (such as bit locker etc.) to encrypt the root volume.

* Additional volumes can be encrypted.


Volumes vs. Snapshots

* Volumes exist on EBS

- Virtual Hard Disk

* Snapshots exist on S3

* You can take a snapshot of a volume, this will store that volume on S3

* Snapshots are point in time copies of Volumes

* Snapshots are incremental, this means that only the blocks that have changed since your last snapshot are moved to S3

* If this is your first snapshot, it may take some time to create


Volumes vs. Snapshots - Security

* Snapshots of encrypted volumes are encrypted automatically

* Volumes restored from encrypted snapshots are encrypted automatically

* You can share snapshots, but only if they are unencrypted.

  - These snapshots can be shared with other AWS accounts or made public


Snapshots of Root Device Volume

* To create a snapshot for Amazon EBS volumes that serve as root devices, you should stop the instance before taking the snapshot.




EBS vs. Instance Store 

* Instance Store Volumes are sometimes called Ephemeral Storage.

* Instance store volumes cannot be stopped. If the underlying host fails, you will lose your data.

* EBS backed instances can be stopped. You will not lose the data on this instance if it is stopped.

* You can reboot both, you will not lose your data.

* By default, both ROOT volumes will be deleted on termination, however with EBS volumes, you can tell AWS to keep the root device volume.


How can I take a snapshot of a RAID Array?

* Problem - Take a snapshot, the snapshot excludes data held in the cache by applications and the OS. This tends not to matter on a single volume, however using multiple volumes in a RAID array, this can be a problem due to interdependencies of the array.


* Solution - Take an application consistent snapshot

- Stop the application from writing to disk

- Flush all chaches to the disk.


- How can we do this?

  Freeze the file system

  Unmount the RAID Array

  Shutting down the associated EC2 instance.

  


Amazon Machine Images 

* AMI's are regional. You can only launch an AMI from the region in which it is stored. However you can copy AMI's to other regions using the console, command line or the Amazon EC2 API.


* Standard Monitoring = 5 Minutes

* Detailed Monitoring = 1 Minute


* CloudWatch is for performance monitoring

* CloudTrail is for auditing


What can I do with Cloudwatch?

* Dashboards - Creates awesome dashboards to see what is happening with your AWS environment

* Alarms - Allows you to set Alarms that notify you when particular thresholds are hit.

* Events - CloudWatch Events helps you to respond to state changes in your AWS resources.

* Logs - CloudWatch Logs helps you to aggregate, monitor, and store logs.


Roles Lab

* Roles are more secure than storing your access key and secret access key on individual EC2 instances.

* Roles are easier to manage

* Roles can be assigned to an EC2 instance AFTER it has been provisioned using both the command line and the AWS console.

* Roles are universal, you can use them in any region.


Instance Meta-data

* Used to get information about an instance (such as public ip)

* curl http://169.254.169.254/latest/meta-data/

* No such thing as user-data for an instance


EFS Features

* Supports the Network File System version 4 (NFSv4) protocol

* You only pay for the storage you use (no pre-provisioning required)

* Can scale up to the petabytes

* Can support thousands of concurrent NFS connections

* Data is stored across multiple AZ's within a region

* Read After Write consistency


What is Lambda?

* AWS Lambda is a compute service where you can upload your code and create a Lambda function. AWS Lambda takes care of provisioning and managing the servers that you use to run the code. You don't have to worry about operating systems, patching, scaling, etc. You can use Lambda in the following ways.


- As an event-driven compute service where AWS Lambda runs your code in response to events. These events could be changes to data in an Amazon S3 bucket or an Amazon DynamoDB table.

- As a compute service to run your code in response to HTTP requests using Amazon API Gateway or API calls made using AWS SDKs. This is what we use at A Cloud Guru





Quiz

- The default region for an SDK is "US-EAST-1"

- AWS SDK supports Python, Ruby, Node.JS, PHO, JAVA (not C++)

- HTTP 5XX is a server side error

- HTTP 4XX is a client side error

- HTTP 3XX is a redirection

- HTTP 2XX is the request was successful

- To find out both private IP address and public IP address of EC2 instance

  => Retrieve the instance Metadata from http://169.254.169.254/latest/meta-data/

- To retrieve instance metadata or userdata you will need to use this IP address

  => http://169.254.169.254

- In order to enable encryption at rest using EC2 and Elastic Block Store you need to

  => Configure encryption when creating the EBS volume

 http://aws.amazon.com/about-aws/whats-new/2014/05/21/Amazon-EBS-encryption-now-available/

- You can have multiple SSL certificates on an Elastic Load Balancer

- Elastic Load Balancers are chargeable

반응형
이전 1 2 3 다음