Safdar Mirza @ Cloud

Webinar: AWS Services for Data Migration

A key part of moving applications to the cloud is migrating data. AWS now offers several simple services for data migration at a petabyte scale. Whether the destination is Amazon S3, Amazon Glacier, Amazon EFS or Amazon EBS, you can move large volumes of data from your facilities to the cloud with AWS. This webinar will explain how you can do this with online or offline transfer services including AWS Storage Gateway, AWS Snowball and Snowball Edge, or EFS File Sync.

 

Watch this webinar to:

  • Learn about AWS services to migrate data sets into AWS file, block and object storage services
  • Determine which AWS service option can fit your requirements for data migration, and
  • See how you can get started. For instance, we demonstrate how you can move block volumes of application data to Amazon EBS by using the AWS Storage Gateway with EBS Snapshots.

Table of Contents

(0:00) Introduction

(1:23) AWS Direct Connect

(2:31) Considerations for moving to the cloud

(5:12) 5 key questions for migrating data to the cloud

(8:47) AWS Snowball (Snow*) family

(11:48) Amazon EFS File Sync

(13:19) Amazon S3 Transfer Acceleration

(16:04) AWS Storage Gateway family introduction

(17:24) File Gateway for moving file data to S3 and for hybrid cloud workloads

(22:38) Tape Gateway for migrating tape backup processes to AWS

(27:50) Volume Gateway for moving block data & creating hybrid cloud storage

(31:55) Demonstration: Migrating block data volumes to Amazon EBS for Amazon EC2 applications using Volume Gateway

(37:34) AWS Partner Network (APN) for Migration & Storage

(38:24) AWS Storage Training

https://www.youtube.com/watch?time_continue=2195&v=hgmFoRf33uA

Original Link: 

https://aws.amazon.com/storagegateway/developer-resources/data-migration-webinar/

 

Getting Started With Concourse on macOS

 

Original Post:
https://medium.com/concourse-ci/getting-started-with-concourse-ci-on-macos-fb3a49a8e6b4

Dockers Machine vs Docker for Mac
https://stories.amazee.io/docker-on-mac-performance-docker-machine-vs-docker-for-mac-4c64c0afdf99

Concourse Github Repository:
https://github.com/concourse/concourse-docker

 

Linux:It uses linux kenrel


Windows/Mac: No Linux Kernel, Docker starts a virtual machine with a small Linux installed and runs Docker containers in there. File system mounts are also not possible natively and need a helper-system in between, which both Docker and Cachalot provide.

Docker Machine:
A set of CLI tools that start a boot2docker virtual machine inside a provided hypervisor (like VirtualBox, Parallels, and VMWare).

Cachalot:


Pre Requests:
New Docker for Mac (HyperKit): Install from following link
https://store.docker.com/


Check Docker Version:


$ docker --version
Docker version 18.03, build c97c6d6

$ docker-compose --version
docker-compose version 1.21.0, build 8dd22a9

$ docker-machine --version
docker-machine version 0.14.0, build 9ba6da9

 

$ docker info

$ wget -nv -O docker-compose.yml https://raw.githubusercontent.com/concourse/concourse-docker/master/docker-compose-quickstart.yml

$ docker-compose up -d

$ docker ps

Check it at :  http://127.0.0.1:8080/

Install Concourse CLI (fly)
https://concourse-ci.org/download.html

File may download as fly.dms - Make sure to rename it to fly (mv fly.dms fly)


$ install fly /usr/local/bin

$ which fly

$ fly -v

$ fly login -t hello -c http://localhost:8080

Note: The -t flag is a target alias and is required for almost every command. This alias comes in very handy when you’re targeting multiple concourse installations.

Setting up a Pipeline:


$ wget -nv https://raw.githubusercontent.com/concourse/testflight/master/pipelines/fixtures/simple.yml

Get more pipeline at following link:


https://github.com/search?l=YAML&p=1&q=org%3Aconcourse+platform%3A&type=Code

$ fly -t hello set-pipeline -p hello-world -c simple.yml


To UnPause pipeline:


$ fly -t hello unpause-pipeline -p hello-world

Trigger pipeline manually:


$ fly -t hello trigger-job -j hello-world/simple

 

Stack

Services

Containers:

Dockerfile:

Dockerfile defines what goes on in the environment inside your container. Access to resources like networking interfaces and disk drives is virtualized inside this environment, which is isolated from the rest of your system, so you need to map ports to the outside world, and be specific about what files you want to “copy in” to that environment. However, after doing that, you can expect that the build of your app defined in this Dockerfile behaves exactly the same wherever it runs.

 

$ docker --version

Docker version 18.03, build c97c6d6

$ docker-compose --version

docker-compose version 1.21.0, build 8dd22a9

$ docker-machine --version

docker-machine version 0.14.0, build 9ba6da9

 

## List Docker CLI commands
docker
docker container --help

## Display Docker version and info
docker --version
docker version
docker info

## Execute Docker image
docker run hello-world

## List Docker images
docker image ls

## List Docker containers (running, all, all in quiet mode)
docker container ls
docker container ls --all
docker container ls -aq

## Stop a container
docker container ls # get CONTAINER ID
docker container stop <CONTAINER ID>

## Remove all containers
docker rm $(docker ps -a -q)


Docker Cheat Sheet

Here is a list of some basic Docker Commands to help you get started with Docker. This helped me a lot and still does from time to time. Note, this not meant to be a complete Docker reference. For complete list, please visit docs.docker.com

Containers

Pull a base image.
docker pull ubuntu

Start a container
docker run --name docker-nginx -p 80:80 -d nginx
docker run --name docker-nginx -p 80:80 -d -v /app/html:/usr/share/nginx/html nginx

For list of running containers
docker ps

For list of all containers
docker ps - a

Restart a Container.
docker restart container_id

Show all running process in a Container.
docker top container_id

Stop container
docker stop container_id

Remove container
docker rm container_id>

Kill containers and remove them
docker stop $(docker ps -aq)
docker rm $(docker ps -aq)
docker rm $(docker kill $(docker ps -aq))

Clean up old containers that are weeks old (via Stackoverflow)
docker ps --filter "status=exited" | grep 'weeks ago' | awk '{print $1}' | xargs --no-run-if-empty docker rm

Print the last 100 lines of a container’s logs 
docker logs --tail 100 container_id

Create a new bash process inside the container and connect it to the terminal
docker exec -it container_id bash

Docker Container link
Docker run -d -p 5000:5000 --link redis dockerapp:v0.3

Docker Images

Remove all images
docker rmi $(docker images -q) -f
docker rmi $(docker images -qf "dangling=true")

Remove all images except “my-image”
You could use grep to remove all except my-image and ubuntu
docker rmi $(docker images | grep -v ‘ubuntu\|my-image’ | awk {‘print $3’})

Save a running container as an image
docker commit -m "commit message" -a "author" container_id username/image_name:tag

Other Docker Commands

Run commands inside an existing container
docker exec -it [containerID] bash

Push to Images to Docker
docker tag 2f98ca2e63ab docker-nginx:1.01
docker login --username muralibala
docker push docker-nginx:1.01

To find Ip address
docker machine ls

List the networks
docker network ls

List the volumes
docker volume ls