Shadow IT exists in your company. Some form of DevOps is being implemented, and somewhere online tools and the public cloud are being used without your approval. Shadow IT can create many risks and bad economic management of IT resources. You can choose to go to war and forbid the use of the new methodologies and tools or learn how to leverage them. This Advisor advocates for the latter.

Today it is legally possible to store credit cards and healthcare patients’ information in the public cloud. This recognition shows that public cloud has matured and is ready for prime time. Embracing the public cloud requires the use of new tools that you might not yet have control over.

Recent methodologies, like DevOps and Agile, give a lot of freedom to people and teams. I recommend that you not try to change that. The kind of IT person who uses cloud and DevOps tends to be skilled, eager to deliver, and educated. You should focus on helping these people with their shortcomings and make sure you understand their tools; be open, understand, lead, and help them on their path to be more productive.

This Advisor provides an introduction to the most used services delivered by AWS (the leading cloud provider). The goal is to share knowledge and trigger your curiosity and will to make things happen. We cover the following AWS services: EC2/VPC (virtual datacenter), S3 (Internet storage), CloudTrail (auditing), and Route 53 (DNS).

Some Warnings About AWS Procurement

AWS is a technology company; it is flexible on technical things and inflexible on procurement and finances. AWS cares about standards — its standards (not yours). Volume discounts usually do not exist with AWS. Large companies buying large quantities receive the same price as a one-person startup. This approach is valid for all AWS services. When you buy from AWS, you pay the full price; if you go through a partner, you can get some discounts.

In AWS’s view, anything nontechnical is better delegated or avoided altogether. As a consequence, AWS asks customers to adapt to its procurement processes, its time scale, and its billing and payment methods. For any well-structured company, this is a problem. You have your procurement rules, made to protect your business and culture. Bending or changing procurement rules adds risk and costs to your cloud adoption. Be ready for such a political and financial fight.

Proposed Approach

AWS is not inventing much; it is offering services that were very expensive at a low price. So as an IT executive, you should grasp the concepts fast.

AWS offering changes on a daily basis, so I’ll focus on the non-moving parts of the services. Each service will include:

• A short description of what the service is about.

• Notes relevant for executives.

• Important points around budget and pricing.

EC2 + VPC

Together, EC2 (Elastic Cloud Computing) and VPC (Virtual Private Cloud) provide a full virtual data center controlled from a centralized Web interface, where you can execute all operations. From setup to configuration, from management to deletion, all is executed from a unified Web interface in real time.

EC2 provides compute, and VPC provides the network. EC2 delivers compute (VM), storage (EBS), load balancers, public IP, and auto scaling. VPC provides networking capabilities, including subnets, firewall, router, NAT, VPN, and DHCP.

Procurement/Budget

A discount is possible. You can buy EC2 on-demand with a pricing per second. You can also book a reserved instance (RI) VM for 12 or 36 months and get a 30% or greater discount. Caution: you always pay a reserved instance, even if it is turned off. No volume discount is available. Running 20,000 VM on-demand costs about 20,000 times the price of a single VM.

Please Remember!

• EC2 provides compute and VPC network. Together they are like a virtual data center.

• Getting a discount by buying a reserved instance requires no technical work; it’s a simple billing concept.

• You should buy reserved instances only for VM that are running 24/7.

• Lots of AWS services are free to use, but their usage creates EC2 elements that cost money. These include CloudFormation, ElasticBeanstalk, Elastic Container Service, and many more.

S3

Simple Storage Service, or S3, is a storage system available online. You can see it as a Network Attached Storage, accessible through a HTTP interface. S3 provides unlimited space and automatic replication. S3 claims a rate of 99.999999999% (11 nines). This means that data saved on S3 is not going to be lost. For use cases not needing quite such a high rate of durability, AWS offers various options, described in the next section.

In AWS, 90% of the data that needs to be saved, communicated, transformed, or shared will use S3. The few exceptions are for services that gather information in real time. For example, detailed billing data is stored on S3, as are snapshots of servers, configuration files, and logs. You get the point. The power of S3 is that it stores data in a reliable way, with a good security mechanism and a well-documented and widely used protocol, HTTP.

An essential S3 word: “bucket.” To store data in S3, you need to save your file in a bucket. The name of a bucket must be globally unique (two customers cannot use the same name). Most security configuration and storage options are managed at the bucket level. It is best practice to have a bucket for each logical segregation of data; for example, projects or level of confidentiality.

Procurement/Budget

S3 pricing is complex and hard to forecast. In S3, you pay for the space you use and also for the amount of data flowing in and out of your S3 instances, the number of API calls, lifecycle transitions, copies between regions, and more. While in most cases the cost of the storage space is orders of magnitude larger than any other cost, it’s important to understand that an application badly architected can easily waste money in an almost invisible way.

S3 has a discount. For up to 50 TB of data, you pay full price; from 50 TB to 450 TB, you get approximately 5% discount; above 450 TB you get another 5%. That’s it. Using half a PB or 2 PB will cost you the same per GB. To overcome this poverty of discounts, AWS provides various levels of quality for your data in S3.

If your data is not accessed very often, you can use the “infrequent access storage” class of storage. This storage is designed for critical, yet non-frequently accessed data. Recent backups fit into this category. You hope your backup data never needs to be reread. This provides the same protection as the standard S3, but cheaper and with a retrieval fee.

The law requires most companies to keep data for longer than the business needs. Data kept for legal purposes needs to be very secure, extremely cheap, and accessible in 24 hours (the state is rarely in a hurry). “Glacier” is made for such use case.

When data can be generated, having a durability of 99.999999999% is overkill. “Reduced redundancy” is available for such cases. It’s cheaper, yet provides 99.99% durability, which is still better than data saved on non-replicated storage systems.

More information on the various classes of S3 is available in the official S3 documentation.

Please Remember

• You pay for the storage, for the flow of data, and many other elements.

• Discounts on S3 are applied automatically.

• Discounts are small.

• S3 provides different classes of storage; each made to match a classical storage use case:

  • Standard — for critical information.

  • Infrequently accessed data — for data that should not need to be accessed often (like recent backups).

  • Glacier — for data required to be kept by law but not by the business (like old backups).

  • Reduced redundancy — for data that can be regenerated.

• You can set up an automated way to move data between storage classes.

CloudTrail

As the AWS documentation website describes it:

AWS CloudTrail is an AWS service that helps you enable governance, compliance, and operational and risk auditing of your AWS account. Actions taken by a user, role, or an AWS service are recorded as events in CloudTrail. Events include actions taken in the AWS Management Console, AWS Command Line Interface, and AWS SDKs and APIs.

CloudTrail is an audit tool that stores enough information for auditors of different specialities (e.g., security, legal, finance, operations) to do their job without interfering with your employees or team. In my experience, this is a life saver, as expensive subject matter experts tend to be hijacked away from their main teams by auditors.

An important concept of CloudTrail is the difference between management events and data events:

• Management events are related to management operations, like the creation of a VM, changes in the security settings, and creation of a logging configuration.

• Data events are more granular and will also list creation of files and execution of lambda function. Data events are turned off by default and, when turned on, can rapidly generate lots of data (and can get out of control).

Procurement/Budget

With CloudTrail, the last 90 days of management data is available for free. If you need to keep the data for longer or create multiple copies of it, you will pay a fee that depends on the volume of actions recorded.

The pricing is done at three levels:

1. It is free to read the log of the management events of the last 90 days.

2. There is a fee to durably store management events logs (the price unit is currently per 100K events).

3. There is a fee to durably store data events logs (also currently priced per 100K events).

4. Logs are saved on S3, for which there is an additional fee.

Please Remember

• CloudTrail doesn’t work with all AWS services. For an updated list, see here.

• CloudTrail records events.

• CloudTrail differentiates between two type of events: management events and data events.

• You can get the last 90 days of management events for free.

Route53

Route53 is the AWS DNS service. It is well integrated with the other AWS services, particularly EC2 and VPC. If you need to create a new service that is going to be implemented with AWS, Route53 can help accelerate the time-to-market, quality-of-service and disaster-recovery capabilities.

Route53 provides three main services:

1. Domain name registration (like GoDaddy, Bluehost, and others).

2. It acts as a standard Internet DNS, routing Internet requests made to your domain name.

3. It can check the health of the pointed resources. This is not standard DNS but is very useful.

Route53 is AWS services–aware, and can for example point to AWS resources directly, like elastic load balancers. Route53 has many routing policy algorithms (i.e., round-robin, weight, geolocation, geoproximity, failover, and latency), allowing for great flexibility.

By mixing the load-balancing algorithm with the health check capability, Route53 becomes a global load balancer. In such a configuration, it is capable of sending users to the closest active Web server, improving the user experience; or it can automatically direct any user to a secondary site in case of a disaster. Route53 is a cheap disaster-recovery solution.

Procurement/Budget

Route53 cost is low; less than US $1 per billion requests per month. There is a fee for registering a new domain, as with any other register company.

Please Remember

• Route53 is a DNS server.

• It can route/distribute traffic using multiple algorithms.

• Route53 allows registration of domains and manages their DNS entries.

• It can perform a health check on the pointed resources.

• Route53 provides some AWS-specific features.

• It can be used as a global load balancer.

Conclusion

As a leader, you understand that the survival of your company depends on IT. You have the hard task to balance business requirements (i.e., money and risks) with technology. Ask your team what they are using to improve IT, discover where cloud is used, ask IT architects to explain their project, go into the financials. Discover how you can help your teams do more.