Skip to content
Go back

05 - AWS Migration & Transfer Services

Published:  at  07:00 AM

AWS Migration & Transfer Services


πŸ“‹ AWS Application Discovery Service

Key PointsDetailed Notes
What is it?Assessment tool for understanding on-premises infrastructure before migration
Main Purposeβ€’ Automatically discovers servers, applications, dependencies
β€’ Maps application relationships and performance metrics
β€’ Provides cost estimation for AWS migration
Deployment Typesβ€’ Agent-based: Install agents on servers for detailed discovery
β€’ Agentless: Network-based discovery with less detail
Key Benefitsβ€’ Automated Discovery: Eliminates manual inventory
β€’ Dependency Mapping: Visualizes application relationships
β€’ Performance Insights: CPU, memory, network utilization
β€’ Risk Reduction: Identifies migration challenges early
Limitationsβ€’ Surface-level discovery: May miss complex dependencies
β€’ 90-day data retention: Limited historical data
β€’ Agent overhead: Can impact system performance
β€’ Containerized workloads: Struggles with dynamic environments
Best Use Casesβœ… Large environments (50+ servers)
βœ… Unknown dependencies
βœ… Cost estimation needs
❌ Well-documented small environments

Simple Real-World Example:

🏦 Trading Platform Migration
Problem: Bank with 500+ servers, unknown dependencies
Solution: Deployed ADS agents to discover infrastructure
Discovery: Trading system connected to 15 databases + 8 middleware
Result: Proper migration sequence, 40% cost savings identified
Timeline: 1 month assessment vs 6 months manual work

Integration Pattern:

ADS β†’ Migration Hub β†’ Cost Explorer β†’ MGN/DMS

πŸš› AWS Application Migration Service (MGN)

Key PointsDetailed Notes
What is it?Automated lift-and-shift migration with minimal downtime
Core Conceptβ€’ Continuous replication from source to AWS
β€’ Test environment before cutover
β€’ Minutes of downtime during final switch
Migration Process1. Install agents on source servers
2. Continuous sync to AWS replicas
3. Test everything in AWS environment
4. Quick cutover during maintenance window
Key Advantagesβ€’ Minimal downtime: Minutes, not hours
β€’ Broad compatibility: Physical, virtual, cloud servers
β€’ Built-in testing: Validate before production
β€’ Rollback capability: Can reverse if needed
Limitationsβ€’ No modernization: Pure lift-and-shift
β€’ Bandwidth requirements: Significant for initial sync
β€’ License costs: May carry over existing licenses
β€’ Agent dependency: Must install on all servers
Perfect Forβœ… Large VM environments (100+ machines)
βœ… 99.9%+ uptime requirements
βœ… Need testing before cutover
❌ Applications needing modernization

Simple Real-World Example:

πŸ›’ E-commerce Platform Migration
Challenge: 200 VMs, Black Friday traffic, 99.95% uptime SLA
Process: Install agents β†’ Continuous replication β†’ Test β†’ Cutover
Results: 
β€’ 12 weeks total (vs 6 months traditional)
β€’ 99.97% uptime achieved
β€’ 40% cost reduction
β€’ Zero data loss

Remember: MGN = Minimal downtime, Great testing, No modernization


πŸ’Ύ AWS Database Migration Service (DMS)

Key PointsDetailed Notes
What is it?Database migration with zero downtime and cross-engine support
Migration Typesβ€’ Homogeneous: Oracle β†’ Oracle (same database type)
β€’ Heterogeneous: Oracle β†’ PostgreSQL (different engines)
β€’ Continuous replication: Ongoing sync for analytics
Core Process1. Source remains online during migration
2. Continuous data sync to target
3. Data validation ensures integrity
4. Cutover when ready with minimal downtime
Key Benefitsβ€’ Zero downtime: Source DB stays operational
β€’ Cross-engine support: Migrate between different databases
β€’ Built-in monitoring: Real-time progress tracking
β€’ Data validation: Automatic integrity checking
Challengesβ€’ Limited transformations: Not for complex data changes
β€’ Learning curve: Requires replication knowledge
β€’ Large objects: Challenges with binary data
β€’ Schema complexity: Manual work for procedures/triggers
Data Engineering Useβ€’ CDC streams: Real-time change capture
β€’ Data lake population: Operational data to S3
β€’ Multi-source consolidation: Combine databases

Simple Real-World Example:

πŸͺ Retail Inventory Migration: Oracle β†’ PostgreSQL
Challenge: 2TB database, 150 stores, zero downtime
Process: Schema conversion β†’ DMS setup β†’ Continuous replication β†’ Cutover
Results:
β€’ 45 minutes downtime (vs 3 hours planned)
β€’ 62% cost reduction
β€’ 15% performance improvement
β€’ All 150 stores operational

πŸ”§ AWS Schema Conversion Tool (SCT)

Key PointsDetailed Notes
What is it?Desktop tool for converting database schemas between engines
Relationship to DMSβ€’ SCT converts schemas (structure, procedures, functions)
β€’ DMS migrates data (actual records)
β€’ Use together for heterogeneous migrations
Conversion Process1. Analyze source database complexity
2. Auto-convert 80-90% of objects
3. Flag manual work needed
4. Generate reports with effort estimates
Key Featuresβ€’ Assessment reports: Complexity analysis
β€’ Automatic conversion: Most database objects
β€’ Cost estimation: AWS target environment costs
β€’ Code conversion: Stored procedures, triggers, functions
Limitationsβ€’ Manual review needed: Complex objects require adjustment
β€’ Desktop dependency: Must install locally
β€’ Version compatibility: Keep updated with DB engines

Simple Conversion Example:

🏭 ERP Migration: SQL Server β†’ PostgreSQL
Analysis: 500 tables, 150 procedures, 25 years of logic
SCT Results:
β€’ 85% automatic conversion
β€’ 15% manual review needed
β€’ 5 weeks vs 6 months manual work
β€’ $180K/year licensing savings

⚑ AWS DataSync

Key PointsDetailed Notes
What is it?Automated online data transfer between on-premises and AWS
Performanceβ€’ 10x faster than traditional tools
β€’ Network optimization and compression
β€’ Parallel transfers for efficiency
Transfer Typesβ€’ One-time: Initial data migration
β€’ Scheduled: Regular backups/updates
β€’ Triggered: Event-based transfers
Key Featuresβ€’ Data validation: Integrity verification
β€’ Bandwidth optimization: Smart compression
β€’ Monitoring: Detailed progress tracking
β€’ Scheduling: Automated recurring jobs
Requirementsβ€’ Stable internet: High-bandwidth connection needed
β€’ DataSync agent: On-premises deployment
β€’ Network access: Proper firewall configuration
Cost Considerationsβ€’ Pay per GB: Can be expensive for large, frequent transfers
β€’ Alternative for huge datasets: Consider Snow Family

Simple Transfer Example:

🎬 Media Company: 50TB Video Library
Challenge: Transfer video content to AWS for global CDN
Process: Deploy agent β†’ Configure transfer β†’ Monitor progress
Results:
β€’ 6 days vs 4 weeks traditional
β€’ 83% cost reduction ($5,200 vs $31,000)
β€’ Automated daily uploads (500GB/day)
β€’ Global content delivery enabled

Decision Matrix:

Data SizeInternet SpeedRecommendation
< 1TBGoodDataSync
1-10TBGoodDataSync
10-80TBPoorSnow Family
> 100TBAnySnow Family

πŸ“¦ AWS Snow Family

Key PointsDetailed Notes
What is it?Physical devices for offline data transfer at petabyte scale
Device Typesβ€’ Snowcone (8TB): Small remote locations
β€’ Snowball Edge (80TB): Standard large transfers
β€’ Snowmobile (100PB): Extreme large datasets
When to Useβ€’ 100+ days to transfer over internet
β€’ $10,000+ in bandwidth costs
β€’ Poor connectivity or security concerns
β€’ Petabyte datasets that break normal tools
Snowball Edge Specialβ€’ Edge computing: Process during transfer
β€’ Local analytics: Work without internet
β€’ Disconnected operations: Remote locations
Process Flow1. Order device from AWS
2. Load data locally
3. Ship back to AWS
4. AWS imports to S3
5. Device wiped and returned
TimelineTotal: 2-3 weeks (including shipping)

Simple Transfer Example:

🧬 Genomics Lab: 200TB Research Data
Challenge: 15 years of DNA data, 2-month deadline, 100 Mbps connection
Solution: 3 Γ— Snowball Edge devices
Process: Order β†’ Load data β†’ Ship β†’ AWS import
Results:
β€’ 8 weeks vs 8+ months network transfer
β€’ $14,200 vs $85,000 traditional approach
β€’ 83% cost reduction
β€’ Zero research downtime

πŸ“ AWS Transfer Family

Key PointsDetailed Notes
What is it?Managed file transfer supporting legacy protocols
Supported Protocolsβ€’ SFTP: Secure FTP (most common)
β€’ FTPS: FTP with SSL security
β€’ FTP: Basic (avoid if possible)
β€’ AS2: Business-to-business standard
Target Storageβ€’ Amazon S3: Primary destination
β€’ Amazon EFS: File system storage
Key Benefitsβ€’ No infrastructure: Fully managed
β€’ Legacy support: Works with old systems
β€’ AWS integration: Direct S3/EFS writes
β€’ Auto-scaling: Handles volume spikes
Cost Structureβ€’ Per endpoint: Monthly endpoint charges
β€’ Per GB transferred: Data transfer costs
β€’ Can be expensive: For high-volume continuous use
Perfect Forβœ… Partner file exchanges
βœ… Legacy system integration
βœ… Compliance requirements
❌ High-volume internal transfers

Simple B2B Example:

🏦 Bank Partner File Exchange
Challenge: 200+ partner banks, 50,000 daily files, legacy SFTP servers
Solution: Transfer Family managed SFTP endpoints
Process: Configure endpoints β†’ Migrate partners β†’ Automate processing
Results:
β€’ 93% cost reduction ($7,410 vs $100,000/month)
β€’ 3x faster file transfers
β€’ Partner onboarding: 5 days vs 30 days
β€’ 24/7 automated operations

🎯 Service Selection Guide

Migration NeedPrimary ServiceSupporting Services
Assessment & PlanningApplication Discovery ServiceMigration Hub, Cost Explorer
Application MigrationApplication Migration Service (MGN)Systems Manager, CloudWatch
Database MigrationDMS + Schema Conversion ToolS3, Redshift, Glue
Large File TransfersDataSync (online) or Snow (offline)S3, Lambda, CloudWatch
Ongoing File ExchangeTransfer FamilyS3, EFS, Lambda
Real-time Data StreamsKinesis Data StreamsLambda, S3, Redshift

πŸ“š Summary Section

Key Migration Journey:

  1. πŸ“‹ ASSESS: Use Application Discovery Service to understand current state
  2. πŸ”„ CONVERT: Use Schema Conversion Tool for database schema changes
  3. πŸš› MIGRATE: Use MGN for applications, DMS for databases
  4. πŸ“¦ TRANSFER: Use DataSync for ongoing files, Snow for massive datasets
  5. πŸ”— INTEGRATE: Use Transfer Family for partner file exchanges

Critical Success Factors:

Common Pitfalls to Avoid:

Quick Reference Decision Tree:

What are you doing?
β”œβ”€β”€ Don't know what you have β†’ Application Discovery Service
β”œβ”€β”€ Moving applications β†’ Application Migration Service (MGN)
β”œβ”€β”€ Moving databases β†’ DMS + Schema Conversion Tool
β”œβ”€β”€ Moving large files β†’ DataSync (online) or Snow (offline) 
└── Ongoing file transfers β†’ Transfer Family

Memory Aids:


πŸ”— Essential Resources for Further Study

Best Practices:


Study Tip: Review the Summary Section regularly and use the memory aids to reinforce learning. Practice with the Decision Tree until service selection becomes automatic.


Suggest Changes

Previous Post
Code smell
Next Post
04 - AWS Database Ecosystem - Complete Service Breakdown