Amazon Web Services AWS Certified Data Engineer Data-Engineer-Associate Question # 8 Topic 1 Discussion

Problem Analysis:

The company needs to migrate both anapplicationand anon-premises Apache Kafka serverto AWS.

Incremental updates from an on-premises Oracle database are processed by Kafka.

The solution must follow areplatform migration strategy, prioritizing minimal changes andlow management overhead.

Key Considerations:

Replatform Strategy: This approach keeps the application and architecture as close to the original as possible, reducing the need for refactoring.

The solution must provide amanaged Kafka serviceto minimize operational burden.

Low overhead solutions like serverless services are preferred.

Solution Analysis:

Option A: Kinesis Data Streams

Kinesis Data Streams is an AWS-native streaming service but is not a direct substitute for Kafka.

This option would require significant application refactoring, which does not align with the replatform strategy.

Option B: MSK Provisioned Cluster

Managed Kafka service with fully configurable clusters.

Provides the same Kafka APIs but requires cluster management (e.g., scaling, patching), increasing management overhead.

Option C: Amazon Kinesis Data Firehose

Kinesis Data Firehose is designed for data delivery rather than real-time streaming and processing.

Not suitable for Kafka-based applications.

Option D: MSK Serverless

MSK Serverless eliminates the need for cluster management while maintaining compatibility with Kafka APIs.

Automatically scales based on workload, reducing operational overhead.

Ideal for replatform migrations, as it requires minimal changes to the application.

Final Recommendation:

Amazon MSK Serverlessis the best solution for migrating the Kafka server and application with minimal changes and the least management overhead.

References:

Amazon MSK Serverless Overview

Comparison of Amazon MSK and Kinesis