블로그 이미지
개발자로서 현장에서 일하면서 새로 접하는 기술들이나 알게된 정보 등을 정리하기 위한 블로그입니다. 운 좋게 미국에서 큰 회사들의 프로젝트에서 컬설턴트로 일하고 있어서 새로운 기술들을 접할 기회가 많이 있습니다. 미국의 IT 프로젝트에서 사용되는 툴들에 대해 많은 분들과 정보를 공유하고 싶습니다.
솔웅

최근에 받은 트랙백

글 보관함

calendar

      1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30    


CloudFormation



What is CloudFormation?




One of the most powerful parts of AWS, CloudFormation allows you to take what was once traditional hardware infrastructure and convert it into code.


CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion.


You don't need to figure our the order for provisioning AWS services or the subtleties of making those dependencies work. CloudFormation takes care of this for you.


After the AWS resources are deployed, you can modify and update them in a controlled and predictable way, in effect applying version control to your AWS infrastructure the same way you do with your software.




CloudFormation Stack vs. Template


A CloudFormation Template is essentially an architectural diagram and a CloudFormation Stack is the end result of that diagram (i.e. what is actually provisioned).


You create, update, and delete a collection of resources by creating, updating, and deleting stacks using CloudFormation templates.


CloudFormation templates are in the JSON format or YAML.



Elements of A Template


Mandatory Elements

- List of AWS Resources and their associated configuration values


Optional Elements

- The template's file format & version number

- Template Parameters

  : The input values that are supplied at stack creation time. Limit of 60

- Output Values

  : The output values required once a stack has finished building (such as the public IP address, ELB address, etc.) Limit of 60.

- List of data tables

  : Used to look up static configuration values such AMI's etc.

  


Outputting Data


- You can use Fn:GetAtt to output data



Exam Tips


- By default, the "automatic rollback on error" feature is enabled

- You are charged for errors

- CloudFormation is free

- Stacks can wait for applications to be provisioned using the "WaitCondition"

- You can use Fn:GetAtt to output data

- Route53 is completely supported. This includes creating new hosted zones or updating existing ones.

- You can create A Records, Aliases etc.

- IAM Role Creation and Assignment is also supported.


1~2 questions in Exam


===========================



Cloud Formation Quiz


- The default scripting language for CloudFormation is : JSON

- Cloud Formation itself is free, however the resources it provisions will be charged at the usual rates. : True

- What happens if Cloud Formation encounters an error by default?

  : It will terminate and rollback all resources created on failure

- You are creating a virtual data center using cloud formation and you need to output the DNS name of your load balancer. What command would you use to achieve this?

  : FN::GetAtt

- What language are cloud formation templates written in? : JSON



======================================


Shared Responsibility Model



===========================


Shared Responsibility Model Quiz


- You are required to patch OS and Applications in RDS? : False

- In the shared responsibility model, what is AWS's responsibility?

  : Restricting access to the data centers, proper destruction of decommissioned disks, patching of firmware for the hardware on which your AWS resources reside



================================




DNS


What is DNS?


If you've used the internet, you've used DNS. DNS is used to convert human friendly domain names (such as http://acloud.guru) into an Internet Protocol (IP) address (such as http://82.124.53.1).


IP addresses are used by computers to identify each other on the network. IP addresses commonly come in 2 different forms, IPv4 and IPv6.



IPv4 vs. IPv6


The IPv4 space is a 32 bit field and has over 4 billion different addresses (4,294,967,296 to be precise).


IPv6 was created to solve this depletion issue and has an address space of 128 bits which in theory is

340,282,366,920,938,463,463,374,607,431,768,211,456 addresses or 340 undecillion addresses



Top Level Domains


If we look at common domain names such as google.com, bbc.co.uk. acloud,guru etc. you will notice a string of characters separated by dots (periods). The last word in a domain name represents the "top level domain". The second word in a domain name is known as a second level domain name (this is optional though and depends on the domain name).

.com, .edu, .gov, .co.uk, .gov.uk, .com.au


These top level domain names are controlled by the Internet Assigned Numbers Authority (IANA) in a root zone database which is essentially a database of all available top level domains. You can view this database by visiting

http://www.iana.org/domains/root/db



Domain Registrars


Because all of the names in a given domain name have to be unique there needs to be a way to organize this all so that domain names aren't duplicated. This is where domain registrars come in. A registrar is an authority that can assign domain names directly under one or more top-level domains. These domains are registered with InterNIC, a service of ICANN, which enforces uniqueness of domain names across the Internet. Each domain name becomes registered in a central database known as the WhoIS database.


Popular domain registrars include GoDaddy.com, 123-reg.co.uk etc.




SOA Records


The SOA record stores information about


- The name of the server that supplied the data for the zone.

- The administrator of the zone.

- The current version of the data file.

- The number of seconds a secondary name server should wait before checking for updates

- The number of seconds a secondary name server should wait before retrying a failed zone transfer

- The maximum number of seconds that a secondary name server can use data before it must either be refreshed or expire.

- The default number of seconds for the time-to-live file on resource records.



NS Records


NS stands for Name Server records and are used by Top Level Domain servers to direct traffic to the Content DNS server which contains the authoritative DNS records.



A Records


An 'A' record is the fundamental type of DNS record and the 'A' in A record stands for 'Address'. The A record is used by a computer to translate the name of the domain to the IP address. For example http://www.acloud.guru might point to http://123.10.10.80



TTL


The length that a DNS record is cached on either the Resolving Server or the users own local PC is equal to the value of the "Time To Live" (TTL) in seconds. The lower the time to live, the faster changes to DNS records take to propagate throughout the internet.



CNAMES


A Canonical Name (CName) can be used to resolve one domain name to another. For example, you may have a mobile website with the domain name http://m.acloud.guru that is used for when users browse to your domain name on their mobile devices. You may also want the name http://mobile.acloud.guru to resolve to this same address.



Alias Records


Alias records are used to map resource record sets in your hosted zone to Elastic Load Balancers, CloudFront distributions, or S3 buckets that are configured as websites.


Alias records work like a CNAME record in that you can map one DNS name (www.example.com) to another 'target' DNS name (elb1234.elb.amazonaws.com).


Key difference - A CNAME can't be used for naked domain names (zone apex record). You can't have a CNAME for http://acloud.guru, it must be either an A record or an Alias.


Alias resource record sets can save you time because Amazon Route 53 automatically recognizes changes in the record sets that the alias resource record set refers to.


For example, suppose an alias resource record set for example.com points to an ELB load balancer at lb1-1234.us-east-1.elb.amazonaws.com. If the IP address of the load balancer changes, Amazon Route 53 will automatically reflect those changes in DNS answers for example.com without any changes to the hosted zone that contains resource record sets for example.com.



Exam Tips


- ELB's do not have pre-defined IPv4 addresses, you resolve to them using a DNS name.

- Understand the difference between an Alias Record and a CNAME.

- Given the choice, always choose an Alias Record over a CNAME





저작자 표시 비영리 동일 조건 변경 허락
신고



SNS (Simple Notification Service)







Amazon Simple Notification Service (Amazon SNS) is a web service that makes it easy to set up, operate, and send notifications from the cloud.


It provides developers with a highly scalable, flexible, and cost-effective capability to publish messages from an application and immediately deliver them to subscribers or other applications.


Amazon SNS follows the "publish-subscribe" (pub-sub) messaging paradigm, with notifications being delivered to clients using a "push" mechanism that eliminates the need to periodically check or "poll" for new information and updates.


With simple APIs requiring minimal up-front development effort, no maintenance or management overhead and pay-as-you-go pricing, Amazon SNS gives developers an easy mechanism to incorporate a powerful notification system with their applications.


Push notifications to Apple, Google, Fire OS, and Windows devices, as well as Android devices in China with Baidu Cloud Push.


Besides pushing cloud notifications directly to mobile devices, Amazon SNS can also deliver notifications by SMS text message or email, to Amazon Simple Queue Service (SQS) queues, or to any HTTP endpoint.


To prevent messages from being lost, all messages published to Amazon SNS are stored redundantly across multiple availability zones.





SNS - Topics



SNS allows you to group multiple recipients using topics. A topic is an "access point" for allowing recipients to dynamically subscribe for identical copies of the same notification.


One topic can support deliveries to multiple endpoint types -- for example, you can group together iOS, Android and SMS recipients. When you publish once to a topic, SNS delivers appropriately formatted copies of your message to each subscriber.






SNS Benefits


- Instantaneous, push-based delivery (no polling)

- Simple APIs and easy integration with applications

- Flexible message delivery over multiple transport protocols

- Inexpensive, pay-as-you-go model with no up-front costs

- Web-based AWS Management Console offers the simplicity of a point-and-click interface




SNS vs. SQS


- Both Messaging Services in AWS

- SNS - Push

- SQS - Polls (Pulls)









SNS Pricing


- Users pay $0.50 per 1 million Amazon SNS Requests

- $0.06 per 100,000 Notification deliveries over HTTP

- $0.75 per 100 Notification deliveries over SMS

- $2.00 per 100,000 Notification deliveries over Email



SNS FAQ



==============


Creating SNS Topic





================



SNS Summary


- Instantaneous, push-based delivery (no polling)

- Protocols include

  : HTTP

  : HTTPS

  : Email

  : Email-JSON

  : Amazon SQS

  : Application

- Messages can be customized for each protocol



====================


SNS Quiz


- SNS is pull based rather than push based? : False

- Which of these is a protocol NOT supported by SNS

  HTTP, HTTPS, Email, Email-JSON, FTP, SQS, Application

  ==> The answer is FTP

- Messages cannot be customized for each protocol used in SNS? : False

- You have a list of subscribers email addresses that you need to push emails out to on a periodic bases. What do you subscribe them to? : A Topic

- You can use SNS in conjunction with SQS to fan a single message out to multiple SQS queues. : True





======================




AWS SWF (Simple Workflow Service)



Amazon Simple Workflow Service (Amazon SWF) is a web service that makes it easy to coordinate work across distributed application components. Amazon SWF enables applications for a range of use cases, including media processing, web application back-ends, business process workflows, and analytics pipelines, to be designed as a coordination of tasks.


Tasks represent invocations of various processing steps in an application which can be performed by executable code, web service calls, human actions, and scripts.



SWF Workers


Workers are programs that interact with Amazon SWF to get tasks, process received tasks, and return the results.



SWF Decider


The decider is a program that controls the coordination of tasks, i.e. their ordering, concurrency, and scheduling according to the application logic.






SWF Workers & Deciders


The workers and the decider can run on cloud infrastructure, such as Amazon EC2, or on machines behind firewalls. Amazon SWF brokers the interactions between workers and the decider. It allows the decider to get consistent views into the progress of tasks and to initiate new tasks in an ongoing manner.


At the same time, Amazon SWF stores tasks, assigns them to workers when they are ready, and monitors their progress. It ensures that a task is assigned only once and is never duplicated. Since Amazon SWF maintains the application's state durably, workers and deciders don't have to keep track of execution state. They can run independently, and scale quickly.




SWF Domains





Your workflow and activity types and the workflow execution itself are all scoped to a domain. Domains isolate a set of types, executions, and task lists from others within the same account.


You can register a domain by using the AWS Management Console or by using the RegisterDomain action in the Amazon SWF API.



The parameters are specified in JavaScript Object Notation (JSON) format.



How Long For workflow?


Maximum Workflow can be 1 year and the value is always measured in seconds.



SWF FAQ



SWF vs SQS


- Amazon SWF presents a task-oriented API, whereas 

  Amazon SQS offers a message-oriented API.

- Amazon SWF ensure that a task is assigned only once and is never duplicated. With Amazon SQS, you need to handle duplicated messages and may also need to ensure that a message is processed only once.

- Amazon SWF keeps track of all the tasks and events in an application. With Amazon SQS, you need to implement your own application-level tracking, especially if your application uses multiple queue.





===========================


SWF Quiz


- SWF consists of a domain, workers an deciders? : True

- Maintaining your application's execution state (e.g. which steps have completed, which ones are running, etc.) is a perfect use case for SWF. : True

- Amazon SWF is useful for automating workflows that include long-running human task (e.g. approvals, reviews, investigations, etc.) Amazon SWF reliably tracks the status of processing steps that run up to several days or months. : True

- In Amazon SWF what is a worker? 

  : Workers are programs that interact with Amazon SWF to get tasks, process received tasks, and return the results

- In Amazon SWF what is a decider

  : The decider is a program that controls the coordination of tasks, i.e. their ordering, concurrency, and scheduling according to the application logic.

  




  

============




Elastic Beanstalk (*** 4~5 questions in the Exam)







- With Elastic Beanstalk, you can deploy, monitor, and scale an application quickly

- It provides developers or end users with the ability to provision application infrastructure is an almost transparent way.

- It has a highly abstract focus towards infrastructure, focusing on components and performance - not configuration and specifications

- It attempts to remove, or significantly simplify infrastructure management, allowing applications to deployed into infrastructure environments easily.





Beanstalk key architecture components


- Applications are the high level structure in beanstalk

- Either your entire application, is one EB application, or

- Each logical component of your application, can be a EB application or a EB environment within an application


- Applications can have multiple environments (Prod, Staging, Dev, V1, V2, V1.1 or functional type (front-end, back-end)

- Environments are either single instance or scalable

- Environments are either web server environments or worker environments


- Application Versions are unique packages which represent versions of apps.

- An application is uploaded to Elastic beanstalk as a application bundle - .zip

- Each application can have many versions 1:M relationship

- Application versions can be deployed to environments within an Application




Elastic Beanstalk Exam Tips


- You can have multiple versions of your applications

- Your applications can be split in to tiers (Web Tier/Application Tier/Database Tier)

- You can update your application

- You can update your configuration

- Updates can be 1 instance at a time, a % of instances or an immutable update

- You pay for the resources that you use, but Elastic Beanstalk is free

- If elastic beanstalk creates your RDS database then it will delete it when you delete your application. If not then the RDS instance stays 

- Know what languages are supported


- Apache Tomcat for Java application

- Apache HTTP Server for PHP applications

- Apache HTTP Server for Python applications

- Nginx or Apache HTTP Server for Node.js applications

- Passenger or Puma for Ruby applications 

- Microsoft IIS 7.5, 8.0, and 8.5 for .NET applications

- JAVA SE

- Docker

- Go




==============================


Elastic Beanstalk Quiz


- Elastic Beanstalk is object based storage. : False

- What languages and development stacks is NOT supported by AWS Elastic Beanstalk?

  : Jetty for jbos application 

- Unlike Cloud Formation, Elastic Beanstalk itself is not free AND you must also pay for the resources it provisions. : False




Elastic Beanstalk FAQ



=====================================



저작자 표시 비영리 동일 조건 변경 허락
신고


Simple Queue Service (SQS) ***






Amazon SQS is a web service that gives you access to a message queue that can be used to store messages while waiting for a computer to process them.

Amazon SQS is a distributed queue system that enables web service applications to quickly and reliably queue messages that one component in the application generates to be consumed by another component. A queue is a temporary repository for messages that are awaiting processing.



Using Amazon SQS, you can decouple the components of an application so they run independently, with Amazon SQS easing message management between components. Any component of a distributed application can store messages in a fail-safe queue.



Messages can contain up to 256 KB (***) of text in any format. Any component can later retrieve the messages programmatically using the Amazon SQS API.



The queue acts as a buffer between the component producing and saving data, and the component receiving the data for processing.




This means the queue resolves issues that arise if the producer is producing work faster than the consumer can process it, or if the producer or consumer are only intermittently connected to the network.



Amazon SQL ensures delivery of each message at least once, and supports multiple readers and writers interacting with the same queue.



A single queue can be used simultaneously by many distributed application components, with no need for those components to coordinate with each other to share the queue.



Amazon SQS is engineered to always be available and deliver messages. One of the resulting tradeoffs is that SQS does not guarantee first in, first out delivery of messages. For many distributed applications, each message can stand on its own, and as long as all messages are delivered, the order is not important.



If your system requires that order be preserved, you can place sequencing information in each message, so that you can reorder the messages when the queue returns them.



To illustrate, suppose you have a number of image files to encode. In an Amazon SQS worker queue, you create an Amazon SQS message for each file specifying the command (jpeg-encode) and the location of the file in Amazon S3.



A pool of Amazon EC2 instances running the needed image processing software does the following





SQS Exam Tips


1. Asynchronously pulls the task messages from the queue

2. Retrieves the named file

3. Processes the conversion

4. Write the image back to Amazon S3

5. Writes a "task complete" message to another queue

6. Delete the original task message

7. Checks for more messages in the worker queue




Autoscaling






- Does not offer FIFO

- 12 hours visibility time out

- Amazon SQS is engineered to provide "at least once" delivery of all messages in its queues. Although most of the time each message will be delivered to your application exactly once, you should design your system so that processing a message more than once does not create any errors or inconsistencies.

- 256kb message size now available

- Billed at 64 kb "chunks"

- A 256kb message will be 4 X 64kb "chunks"




SQL Pricing


- First 1 million Amazon SQS Requests per month are free

- $0.50 per 1 million Amazon SQS Requests per month thereafter ($0.00000050 per SQS Request)

- A single request can have from 1 to 10 messages, up to a maximum total payload of 256KB.

- Each 64KB 'chunk' of payload is billed as 1 request. For example, a single API call with a 256KB payload will be billed as four requests.





=========================================


SQS Developer Exam Tips


SQS - Delivery


  SQS Messages can be delivered multiple times and in any order.



SQS - Default Visibility Time Out


  Default Visibility Time Out is 30 seconds


  Maximum Time Out is 12 Hours



When you receive a message from a queue and begin processing it, you may find the visibility timeout for the queue is insufficient to fully process and delete that message. To give yourself more time to process the message, you can extend its visibility timeout by using the ChangeMessageVisibility action to specify a new timeout value. Amazon SQS restarts the timeout period using the new value.





SQS Long Polling


SQS long polling is a way to retrieve messages from your SQS queues. While the traditional SQS short polling returns immediately, even if the queue being polled is empty, SQS long polling doesn't return a response until a message arrives in the queue, or the long poll times out. SQS long polling makes it easy and inexpensive to retrieve messages from your SQS queue as soon as they are available.


Maximum Long Poll Time Out = 20 seconds





Example Questions


Polling in a tight loops is burning CPU cycles and costing the company money. How would you fix this? - To enable the long polling



SQS - Fanning Out


Create an SNS topic first using SNS. Then create and subscribe multiple SQS queues to the SNS topic.


Now whenever a message is sent to the SNS topic, the message will be fanned out to the SQS queues, i.e. SNS will deliver the message to all the SQS queues that are subscribed to the topic.




==========================




SQS Quiz


- SQS was the first service on the AWS platform? - true

- How large can an SQS message be? - 256kb

- What is the default visibility time out setting? - 30 seconds

- An SQS message can be delivered multiple times - True

- You are designing a new application which involves processing payments and delivering promotional emails to customers. You plan to use SQS to help facilitate this. You need to ensure that the payment process takes priority over the creation and delivery of emails. What is the best way to achieve this.

  : Use 2 SQS queues for the platform. Have the EC2 fleet poll the payment SQS queue first. If this queue is empty, then poll the promotional emails queue.

- Your EC2 instance download jobs from the SQS queue, however they are taking too long to process them. What API call can you use to extend the length of time to process the jobs? : ChangeMessageVisibility

- What is the default visibility time out? : 30 seconds

- You have a fleet of EC2 instances that are constantly polling empty SQS queues which is burning CPU compute cycles and costing your company money. What should you do?

  : Enable SQS Long Polling

- What is the maximum long poll time out : 20 seconds

- What amazon service can you use in conjunction with SQS to 'fan out' SQS messages to multiple queues : SNS



========================================





저작자 표시 비영리 동일 조건 변경 허락
신고