sns-sqs-big-payload - GitLite

KO | EN

gitlite — search

#javascript #hacktoberfest #llm #go #machine-learning #rust #nodejs #cli #vue #pytorch #mcp #ios #openai #open-source

sns-sqs-big-payload

(★ 51)

Amazon SNS/SQS client library that enables sending and receiving messages with payload larger than 256KiB via Amazon S3.

파일 탐색기

최종 버전 다운로드 (.zip)

- - build.yml
  - publish.yml
- .prettierrc
- usage-in-lambda.md
- wait-for-url.js
- constants.ts
- index.ts
- sns-producer.ts
- sqs-consumer.ts
- sqs-producer.ts
- types.ts
- util.ts
- sns-sqs.spec.ts
.editorconfig
.gitignore
.npmignore
.prettierrc
docker-compose.yml
LICENSE
package.json
README.md
tsconfig.json
yarn.lock

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/aspecto-io/sns-sqs-big-payload

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd sns-sqs-big-payload

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Docker

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Docker Desktop 컨테이너를 빌드하고 실행하려면 필요합니다. 설치 후 실행해서 백그라운드에 켜두세요.

TMPDIR=/private$TMPDIR docker-compose up

compose 설정 파일에 정의된 서비스들을 대상으로 명령을 실행합니다.

✅ 터미널에 docker compose ps 를 입력해 컨테이너들이 Up 상태인지 확인하세요. README에 포트 번호가 적혀있다면 브라우저에서 http://localhost:포트번호 로 접속해보세요.

이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.

3. Node.js

쉬움

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Node.js npm 명령어를 쓰려면 Node.js가 설치되어 있어야 합니다. LTS(장기지원) 버전을 추천합니다.

npm install sns-sqs-big-payload

package.json에 명시된 라이브러리들을 내려받아 설치합니다.

npm test

이 명령어를 터미널에 그대로 입력해 실행하세요.

✅ 명령어 실행 후 터미널에 나타나는 주소(보통 http://localhost:3000 형태)를 브라우저에서 열어보세요.

이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

![Build](https://github.com/aspecto-io/sns-sqs-big-payload/workflows/Build/badge.svg) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](http://makeapullrequest.com) [![TypeScript](https://badgen.net/npm/types/env-var)](http://www.typescriptlang.org/) [![NPM version](https://img.shields.io/npm/v/sns-sqs-big-payload.svg)](https://www.npmjs.com/package/sns-sqs-big-payload)

# sns-sqs-big-payload

SQS/SNS producer/consumer library. Provides an ability to pass payloads though s3.

## Motivation

[Aspecto](https://www.aspecto.io/?utm_source=github&utm_medium=sqs-sns-big-payload&utm_campaign=readme-p1&utm_content=v1) helps modern development teams solve production issues before they evolve. We collect real production data and perform deep API analysis over it to autogenerate tests and monitor services stability. As a result, we often need to handle large payloads which can't be used with SQS & SNS due to the hard size limit. This library was developed to overcome this challenge - it enables you to manage Amazon SNS & SQS message payloads with Amazon S3 when dealing with payloads larger than 256KB. Key functionality includes:

-   Controlling whether message payloads are always stored in Amazon S3 or only when a message's size exceeds 256KB.
-   Send a message that references a single message object stored in an Amazon S3 bucket.
-   Get the corresponding message object from an Amazon S3 bucket.
-   Handle the interface for large messages between SNS to SQS via S3 bucket in the middle

## Installation

```
npm install sns-sqs-big-payload
```

Important:

> Make sure you also have `aws-sdk` installed, because it's listed as a peer dependency, so won't be installed automatically.

## Usage

The library exports 3 clients:

-   SNS producer
-   SQS producer
-   SQS consumer

All 3 clients are under the same repository since they share a similar contract when sending payloads via S3.

### SNS Producer

```ts
import { SnsProducer } from 'sns-sqs-big-payload';

const snsProducer = SnsProducer.create({
    topicArn: '<topic-arn>',
    region: 'us-east-1',
    // to enable sending large payloads (>256KiB) though S3
    largePayloadThoughS3: true,
    // Opt-in to enable compatibility with
    // Amazon SQS Extended Client Java Library (and other compatible libraries).
    // see https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-s3-messages.html
    extendedLibraryCompatibility: boolean,
    s3EndpointUrl: '...',
});

await snsProducer.sendJSON({
    // ...
});
```

### SQS Producer

```ts
import { SqsProducer } from 'sns-sqs-big-payload';

const sqsProducer = SqsProducer.create({
    queueUrl: '...',
    region: 'us-east-1',
    // to enable sending large payloads (>256KiB) though S3
    largePayloadThoughS3: true,
    // Opt-in to enable compatibility with
    // Amazon SQS Extended Client Java Library (and other compatible libraries).
    // see https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-s3-messages.html
    extendedLibraryCompatibility: boolean,
    s3Bucket: '...',
});

await sqsProducer.sendJSON({
    // ...
});
```

### SQS Consumer

```ts
import { SqsConsumer, SqsConsumerEvents } from 'sns-sqs-big-payload';

const sqsConsumer = SqsConsumer.create({
    queueUrl: '...',
    region: 'us-east-1',
    // to enable loading payloads from S3 automatically
    getPayloadFromS3: true,
    s3Bucket: '...',
    // if the queue is subscribed to SNS
    // the message will arrive wrapped in sns envelope
    // so we need to unwrap it first
    transformMessageBody: (body) => {
        const snsMessage = JSON.parse(body);
        return snsMessage.Message;
    },
    // if you expect json payload - use `parsePayload` hook to parse it
    parsePayload: (raw) => JSON.parse(raw),
    // message handler, payload already parsed at this point
    handleMessage: async ({ payload }) => {
        // ...
    },
    // Opt-in to enable compatibility with
    // Amazon SQS Extended Client Java Library (and other compatible libraries).
    // see https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-s3-messages.html
    extendedLibraryCompatibility: boolean,
});

// to subscribe for events
sqsConsumer.on(SqsConsumerEvents.messageProcessed, () => {
    // ...
});

sqsConsumer.start();

// to stop processing
sqsConsumer.stop();
```

-   The queue is polled continuously for messages using long polling.
-   Messages are deleted from the queue once the handler function has completed successfully.
-   Throwing an error (or returning a rejected promise) from the handler function will cause the message to be left on the queue. An SQS redrive policy can be used to move messages that cannot be processed to a dead letter queue.
-   By default messages are processed by 10 at a time – a new batch won't be received until the previous one is processed. To adjust number of messages that is being processed in parallel, use the `batchSize` option detailed below.

## Usage in lambda

If you have a lambda function subscribed to sqs queue, you can use SqsConsumer in this case too. [This is a short guide.](./docs/usage-in-lambda.md)

## Compatibility with libraries in other languages

If you turn on `extendedLibraryCompatibility`, then the library will be compatible with:

-   Amazon SQS Extended Client Library for Java (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-s3-messages.html)
-   Boto3 Sqs Extended Client Lib for python (https://github.com/timothymugayi/boto3-sqs-extended-client-lib)
-   other libraries that are compatible to the above

Please be careful: This mode is not compatible with the standard mode due to differences in s3 payload.

## Credentials

By default the consumer will look for AWS credentials in the places [specified by the AWS SDK](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/configuring-the-jssdk.html#Setting_AWS_Credentials). The simplest option is to export your credentials as environment variables:

```sh
export AWS_SECRET_ACCESS_KEY=...
export AWS_ACCESS_KEY_ID=...
```

If you need to specify your credentials manually, you can use a pre-configured instance of the [AWS SQS](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SQS.html) client:

```ts
import { SqsConsumer } from 'sns-sqs-big-payload';
import * as aws from 'aws-sdk';

aws.config.update({
    region: 'us-east-1',
    accessKeyId: '...',
    secretAccessKey: '...',
});

const consumer = SqsConsumer.create({
    queueUrl: 'https://sqs.us-east-1.amazonaws.com/account-id/queue-name',
    handleMessage: async (message) => {
        // ...
    },
    sqs: new aws.SQS(),
});

consumer.start();
```

## Events and logging

SqsConsumer has an internal [EventEmitter](https://nodejs.org/api/events.html), you can subscribe for events like this:

```ts
sqsConsumer.on(SqsConsumerEvents.messageProcessed, () => {
    // ...
});
```

It sends the following events:

| Event                     | Params                 | Description                                                                         |
| ------------------------- | ---------------------- | ----------------------------------------------------------------------------------- |
| started                   | None                   | Fires when the polling is started                                                   |
| message-received          | `message`              | Fires when a message is received (one per each message, not per batch)              |
| message-processed         | `message`              | Fires after the message is successfully processed and removed from the queue        |
| batch-processed           | None                   | Fires after the current batch of messages is processed.                             |
| poll-ended                | None                   | Fires after the polling cycle is ended. Useful for graceful shutdown.               |
| stopped                   | None                   | Fires when the polling stops                                                        |
| error                     | `{err, message}`       | Fires in case of processing error                                                   |
| s3-payload-error          | `{err, message}`       | Fires when an error occurs during downloading payload from s3                       |
| s3-extended-payload-error | `{err, message}`       | Fires when a payload from s3 using extended compatibility is not in expected format |
| processing-error          | `{err, message}`       | Fires when an error occurs during processing (only inside `handleMessage` function) |
| connection-error          | `err`                  | Fires when a connection error occurs during polling (retriable)                     |
| payload-parse-error       | `err`                  | Fires when a connection error occurs during parsing                                 |

You can also use this enum if you're using TypeScript

```ts
enum SqsConsumerEvents {
    started = 'started',
    messageReceived = 'message-received',
    messageProcessed = 'message-processed',
    batchProcessed = 'batch-processed',
    pollEnded = 'poll-ended',
    stopped = 'stopped',
    error = 'error',
    s3PayloadError = 's3-payload-error',
    s3ExtendedPayloadError = 's3-extended-payload-error',
    processingError = 'processing-error',
    connectionError = 'connection-error',
    payloadParseError = 'payload-parse-error',
}
```

You may subscribe to those events to add logging for example.

## Testing

Since this library heavily relies on AWS APIs, it is less relevant to run an isolated test using mocks. As a result, we recommend testing it using a [localstack](https://github.com/localstack/localstack) or by using real SQS queues and SNS topics.

To run localstack on mac:

```sh
TMPDIR=/private$TMPDIR docker-compose up
```

To run unit tests:

```sh
npm test
```