This example shows how to build a serverless messaging pipeline using AWS SAM. The project uses SNS to publish messages, SQS queues to separate concerns, and Lambda functions to process the data. It also uses filter policies to control message delivery to each queue.

This architecture is useful when you want to send different types of data to different consumers, and keep your application components decoupled.

Project Structure

The template defines:

  • SNS Topic: Central topic that receives all messages.
  • Two SQS Queues:
    • One for log messages (AppCallLogsQueue)
    • One for conversation messages (AppCallConversationQueue)
  • Filter Policies on each SNS subscription to make sure each queue receives only the relevant messages.
  • Two Lambda Functions:
    • One to process log messages.
    • One to process conversation messages.

How It Works

  1. Publishing Messages:

    • A component (for example, another Lambda or app) publishes a message to the SNS Topic.
    • The message includes an attribute queue with the value "log" or "conversation".

  2. Message Filtering:

    • SNS Subscriptions use filter policies to decide which message goes to which SQS queue.
    • For example:
      • Messages with queue = "log" go to AppCallLogsQueue
      • Messages with queue = "conversation" go to AppCallConversationQueue

  3. Processing Messages:

    • Each queue triggers a different Lambda function.
    • The function receives a batch of messages, processes them, and deletes them from the queue.

This design helps split responsibilities in your system. Each Lambda only focuses on the specific type of message it expects.

Use Cases:

  • Multi-service communication in microservice architectures.
  • Message routing based on type.
  • Asynchronous processing pipelines.

Project Repository: https://github.com/OmarMakled/sam-sns-sqs