{"id":2646,"date":"2019-05-01T19:26:59","date_gmt":"2019-05-01T17:26:59","guid":{"rendered":"http:\/\/miro.borodziuk.eu\/?p=2646"},"modified":"2020-03-10T22:34:32","modified_gmt":"2020-03-10T21:34:32","slug":"elastic-block-store","status":"publish","type":"post","link":"http:\/\/miro.borodziuk.eu\/index.php\/2019\/05\/01\/elastic-block-store\/","title":{"rendered":"Elastic Block Store"},"content":{"rendered":"<p>Elastic Block Store (EBS) is a storage service that creates and manages volumes based on four underlying storage types. Volumes are persistent, can be attached and removed from EC2 instances, and are replicated within a single AZ.<\/p>\n<p><!--more--><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2647\" src=\"http:\/\/miro.borodziuk.eu\/wp-content\/uploads\/EBS.jpg\" alt=\"\" width=\"623\" height=\"353\" srcset=\"http:\/\/miro.borodziuk.eu\/wp-content\/uploads\/EBS.jpg 623w, http:\/\/miro.borodziuk.eu\/wp-content\/uploads\/EBS-300x170.jpg 300w\" sizes=\"(max-width: 623px) 100vw, 623px\" \/><\/p>\n<p>Volume Types<\/p>\n<ul>\n<li>Mechanical scl and stl; Solid State gp2 and iol<\/li>\n<li>scl: Lowest cost, infrequent access, can&#8217;t be boot volume<\/li>\n<li>stl: Low cost, throughput intensive, can&#8217;t be a boot volume<\/li>\n<li>gp2: Default, balance of IOPS\/MiBis &#8211; burst pool IOPS per GB<\/li>\n<li>iol: Highest performance, can adjust size and IOPS separately<\/li>\n<\/ul>\n<p>To protect against AZ failure, EBS snapshots (to S3) can be used. Data is replicated across AZs in the region and (optionally) internationally.<\/p>\n<p>&nbsp;<\/p>\n<p>Legacy non-EBS-optimized instances used a shared networking path for data and storage communications.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2661\" src=\"http:\/\/miro.borodziuk.eu\/wp-content\/uploads\/EBSnon-optimized.jpg\" alt=\"\" width=\"559\" height=\"231\" srcset=\"http:\/\/miro.borodziuk.eu\/wp-content\/uploads\/EBSnon-optimized.jpg 559w, http:\/\/miro.borodziuk.eu\/wp-content\/uploads\/EBSnon-optimized-300x124.jpg 300w\" sizes=\"(max-width: 559px) 100vw, 559px\" \/><\/p>\n<p>EBS-optimized mode, which was historically optional and is now the default, adds optimizations and dedicated communication paths for storage and traditional data networking. This allows consistent utilization of both \u2014 and is one required feature to support higher performance storage.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2662\" src=\"http:\/\/miro.borodziuk.eu\/wp-content\/uploads\/EBSoptimized.jpg\" alt=\"\" width=\"542\" height=\"199\" srcset=\"http:\/\/miro.borodziuk.eu\/wp-content\/uploads\/EBSoptimized.jpg 542w, http:\/\/miro.borodziuk.eu\/wp-content\/uploads\/EBSoptimized-300x110.jpg 300w\" sizes=\"(max-width: 542px) 100vw, 542px\" \/><\/p>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Key Features<\/span><\/p>\n<ul>\n<li>Maximum volume size of 16 TiB Volumes can only be mounted to one instance at a time<\/li>\n<li>Multiple volumes can be mounted to a single instance<\/li>\n<li>Raw, unformatted block storage<\/li>\n<\/ul>\n<p>&#8211; A file system must be created<\/p>\n<p>&#8211; RAID is supported if the OS supports it<\/p>\n<ul>\n<li>Created in an Availability Zone and must be used with instances in that zone<\/li>\n<\/ul>\n<p>&#8211; Automatically replicated in that zone<\/p>\n<p>&#8211; Protects against hardware failure, not high availability<\/p>\n<p>&nbsp;<\/p>\n<p>Persistence<\/p>\n<ul>\n<li>Attached volumes are independent of the instance<\/li>\n<\/ul>\n<p>&#8211; Terminating the instance will not terminate the volume<\/p>\n<ul>\n<li>Root volumes, by default, terminate with the instance<\/li>\n<\/ul>\n<p>&#8211; This behavior is changed by setting<code> DeleteOnTermination<\/code> attribute to false<\/p>\n<p>&nbsp;<\/p>\n<p>Snapshots<\/p>\n<ul>\n<li>Images or backups of EBS volumes<\/li>\n<li>Stored in an Amazon-managed S3 bucket (charged based on volume&#8217;s total size)<\/li>\n<li>Exact copy of the original volume, Encryption included<\/li>\n<li>Incremental in nature, but full volume can be restored from any snapshot<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><strong>EBS snapshots<\/strong> are a point-in-time backup of an EBS volume stored in 53. The initial snapshot is a full copy of the volume. Future snapshots only store the data changed since the last snapshot.<\/p>\n<p>Snapshots can be used to create new volumes and are a great way to move or copy instances between AZs. When creating a snapshot of the root\/boot volume of an instance or busy volume, it&#8217;s recommended the instance is powered off, or disks are &#8220;flushed.&#8221;<\/p>\n<p>Snapshots can be copied between regions, shared, and automated using Data Lifecycle Manager (DLM).<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2652\" src=\"http:\/\/miro.borodziuk.eu\/wp-content\/uploads\/EBSSnapshot.jpg\" alt=\"\" width=\"619\" height=\"477\" srcset=\"http:\/\/miro.borodziuk.eu\/wp-content\/uploads\/EBSSnapshot.jpg 619w, http:\/\/miro.borodziuk.eu\/wp-content\/uploads\/EBSSnapshot-300x231.jpg 300w\" sizes=\"(max-width: 619px) 100vw, 619px\" \/><\/p>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Performance<\/span><\/p>\n<p>EBS uses IOPS (input\/output operations per second) as a performance measure, measured in KiB (kibibytes = 1024 bytes).<\/p>\n<p><em>Burst Buckets:\u00a0 <\/em>Allows an EBS volume to &#8220;burst&#8221; above the baseline performance Volumes earn credits Credits are then spent whenever the volume needs more performance There is a maximum number of credits<\/p>\n<p>\u2022 Not available for Provisioned IOPS SSD (I01)<\/p>\n<p>\u2022 Reported as a <code>BurstBalance<\/code> metric in CloudWatch<\/p>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #999999;\"><strong>SSD<\/strong> <\/span><\/p>\n<p>Consistent performance with sequential and random operations<\/p>\n<p>Good for frequent read\/writes using small I\/O sizes (10PS) IOPS measured with 256 KiB I\/O size (chunks of data)<\/p>\n<p>\u2022 One 1024 KiB operation = 4 IOPS<\/p>\n<p>\u2022 Four 64 KiB sequential operations = 1 IOPS<\/p>\n<p>\u2022 Four 64 KiB random operations = 4 IOPS<\/p>\n<p>&nbsp;<\/p>\n<p><strong>General Purpose SSD Volumes ( gp2 )<\/strong><\/p>\n<ul>\n<li>Volume size: 1 GiB to 16 TiB<\/li>\n<li>Baseline performance: 100 to 10,000 IOPS<\/li>\n<li>3 IOPS per GiB of volume size<\/li>\n<li>Minimum of 100 IOPS (below 33.3 GiB volume size)<\/li>\n<li>Maximum throughput = 160 MiB\/s<\/li>\n<li>Burst Bucket<\/li>\n<\/ul>\n<p>\u2022 5.4 million credits to start (max 3,000 IOPS for 30 minutes)<\/p>\n<p>\u2022 Credits earned at 3 IOPS per GiB of volume size<\/p>\n<p>\u2022 Maximum credits = 5.4 million<\/p>\n<p>\u2022 Volumes greater than 1,000 GiB never deplete (baseline = burst max)<\/p>\n<p>&nbsp;<\/p>\n<p><strong>Provisioned IOPS SSD volumes ( iol )<\/strong><\/p>\n<ul>\n<li>Volume size: 4 GiB to 16 TiB<\/li>\n<li>Baseline performance: 100 to 64,000 IOPS<\/li>\n<li>User\/provisioner chooses a consistent IOPS rate<\/li>\n<li>AWS SLA = 99.9 of the time within 10% of provisioned IOPS<\/li>\n<li>Maximum ratio is 50:1<\/li>\n<li>A 640 GiB volume size or greater can use maximum IOPS<\/li>\n<li>AWS recommends a ratio larger than 2:1 with provisioned IOPS volumes<\/li>\n<li>If 4,000 IOPS are needed, volume size should be less than 2,000 GiB<\/li>\n<li>Maximum throughput = 500 MiB\/s<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #999999;\"><strong>HDD <\/strong><\/span><\/p>\n<p>Optimal performance with large and sequential operations<br \/>\nGood for large streaming workloads (throughput) IOPS measured with 1024 KiB I\/O size (chunks of data)<\/p>\n<p>\u2022 One 1024 KiB operation = 1 IOPS<\/p>\n<p>\u2022 Eight 128 KiB sequential operations = 1 IOPS<\/p>\n<p>\u2022 Eight 128 KiB random operations = 8 IOPS<\/p>\n<p>&nbsp;<\/p>\n<p><strong>Throughput Optimized HDD volumes (<span style=\"color: #999999;\"> stl<\/span> )<\/strong><\/p>\n<ul>\n<li>Not supported as a boot device<\/li>\n<li>Ideal for frequently accessed and throughput intensive workloads Volume size: 500 GiB to 16 TiB<\/li>\n<li>Maximum throughput = 500 MiB\/s<\/li>\n<li>Burst bucket: Credits gained at 40 MiB\/s per TiB; Credit capacity = 1 TiB; Maximum burst = 500 MiB\/s (volume sizes 2 TiB and larger)<\/li>\n<\/ul>\n<p><strong>Cold HDD volumes ( <span style=\"color: #999999;\">scl<\/span> )<\/strong><\/p>\n<ul>\n<li>Not supported as a boot device Ideal for infrequently accessed data and lowest storage cost<\/li>\n<li>Volume size: 500 GiB to 16 TiB<\/li>\n<li>Maximum throughput = 250 MiB\/s Burst bucket<\/li>\n<li>Credits gained at 12 MiB\/s per TiB<\/li>\n<li>Credit capacity = 1 TiB \u2022 Maximum burst = 250 MiB\/s (volume sizes 2 TiB and larger)<\/li>\n<\/ul>\n<p><strong>Magnetic volumes<\/strong><\/p>\n<ul>\n<li>Somewhat deprecated (Previous Generation volume)<\/li>\n<li>Low-cost storage for small volume sizes<\/li>\n<li>Volume Size: 1 GiB to 1 TiB<\/li>\n<li>Burst capability to hundreds of IOPS<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p>Join multiple gp2, lot st1, or sc1 volumes together in a RAID 0 configuration (stripe set) to use the available bandwidth, improving throughput.<\/p>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Cloudwatch Metrics<\/span><\/p>\n<ul>\n<li>Five-minute period data available at no charge<\/li>\n<li>Provisioned IOPS SSD send one-minute period data automatically<\/li>\n<li>Included metrics measure disk management<\/li>\n<\/ul>\n<p>\u2022 <code>VolumeReadBytes, VolumeWriteBytes<\/code><\/p>\n<p>\u2022 <code>VolumeReadOps, VolumeWriteOps<\/code><\/p>\n<p>\u2022 <code>VolumeTotalReadTime, VolumeTotalWriteTime<\/code><\/p>\n<p>\u2022 <code>VolumeIdleTime<\/code><\/p>\n<p>\u2022<code> VolumeQueueLength<\/code><\/p>\n<ul>\n<li>Provisioned IOPS SSD (i01) volumes only:<\/li>\n<\/ul>\n<p>\u2022 <code>VolumeThroughputPercentage <\/code><\/p>\n<p>\u2022 <code>VolumeConsumedReadWri teOps<\/code><\/p>\n<ul>\n<li>General Purpose SSD (gp2), Throughput Optimized HDD (st1), and Cold HDD (sc1) volumes only:<\/li>\n<\/ul>\n<p>\u2022 <code>BurstBalance<\/code><\/p>\n<p>&nbsp;<\/p>\n<p>EBS Status Checks<\/p>\n<ul>\n<li>Tests run every five minutes<\/li>\n<li>Returns: OK, warning, impaired, insufficient-data<\/li>\n<li>User can change the result of the impaired response<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Resizing or changing root volumes<\/span><\/p>\n<p>Modify the size, IOPS, or type of an EBS volume<br \/>\nThe Manual Method:<\/p>\n<ol>\n<li>Modify the EBS volume<\/li>\n<li>Extend the partition to fill available space<\/li>\n<li>Expand the filesystem in the resized partition<\/li>\n<\/ol>\n<p>For Nitro-based instances (e.g. t3.micro):<\/p>\n<pre class=\"lang:sh decode:true\">lsblk\r\ndf -h\r\nsudo file -s \/dev\/nvme?n*\r\nsudo growpart \/dev\/nvmeenl 1\r\nsudo xfs_growfs -d \/<\/pre>\n<p>For T2 instances:<\/p>\n<pre class=\"lang:sh decode:true\">sudo file -s \/dev\/xvd*\r\nsudo growpart \/dev\/xvda 1\r\nsudo resize2fs \/dev\/xvda\r\n<\/pre>\n<p>&nbsp;<\/p>\n<p>The &#8220;Automated&#8221; Method:<\/p>\n<ul>\n<li>We can replace the launch configuration of an Auto Scaling group.<\/li>\n<li>In a true n-tier application that is decoupled, we should then be able to terminate instances one by one to recreate them using the new configuration.<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Ensuring data durability upon EC2 instance termination<\/span><\/p>\n<ul>\n<li>By default, instance store and EBS root volumes are not backed up<\/li>\n<li>Will not persist upon termination<\/li>\n<li>Cannot stop instance store volumes, so termination is the only option<\/li>\n<li>This is why EBS volumes are recommended<\/li>\n<li>How do we save the data on a root volume?<\/li>\n<\/ul>\n<p>&#8211; Uncheck &#8220;Delete on Termination&#8221; in the console: Also a CLI parameter with run-instances<\/p>\n<p>&#8211; Create a snapshot before deletion<\/p>\n<p>&#8211; We can create a separate volume and attach to the instance:\u00a0 Attached volumes persist when the instance is terminated<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Elastic Block Store (EBS) is a storage service that creates and manages volumes based on four underlying storage types. Volumes are persistent, can be attached and removed from EC2 instances, and are replicated within a single AZ.<\/p>\n","protected":false},"author":1,"featured_media":2655,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[77],"tags":[],"_links":{"self":[{"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/posts\/2646"}],"collection":[{"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/comments?post=2646"}],"version-history":[{"count":6,"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/posts\/2646\/revisions"}],"predecessor-version":[{"id":3342,"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/posts\/2646\/revisions\/3342"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/media\/2655"}],"wp:attachment":[{"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/media?parent=2646"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/categories?post=2646"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/tags?post=2646"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}