AWS Serverless FTP File Processing Service

2021-12-28 · 4 min

A common business task is file processing: fetch a file from somewhere like an FTP server, read and transform it, and store the results in a database. There are dozens of ways to do it, from batch scripts to Windows services. This post takes a modern approach using AWS.

The scenario

Acme Corporation needs a file processing system:

• It picks up CSV files from an FTP server and reads the data.
• Some fields in each file need checking against an external API.
• The CSV files have had formatting problems before, so the system should flag any file it can't process for manual inspection, and carry on processing the others.
• Over Christmas, file volumes rise. The system must process more files in the same amount of time.

Polling and transfer

Here's the design, and what each part is for.

1. Lambda (file fetch)

This is the poller. It connects to the FTP server on a schedule, checks a directory for files, downloads them and puts them in an S3 bucket. Nothing else:

• No processing. A Lambda can only run for 15 minutes, and we don't want it stuck downloading and processing files one after another.
• Tighter security. Doing one job means its execution policy only needs access to the resources that job uses.

2. CloudWatch

Because this Lambda only polls and downloads, its CloudWatch logs only contain those I/O events, which makes them easier to follow. Alarms can alert people to problems through SNS or email, depending on requirements. Beyond that, it's simply very useful for tracking events and debugging.

3. S3 (file bucket)

The bucket is a temporary store for downloaded files, and a key part of making the service distributed and resilient. Files reach it through a VPC endpoint, so they never cross the public internet. Two S3 features do a lot of the work:

• Lifecycle policy. Files are deleted automatically to save cost. Agree the retention period with the data provider or your company, since files may contain PII. While a file is kept, a processing failure doesn't mean downloading it again. Files on FTP servers can be short-lived or overwritten, so this also avoids having to ask for them again.
• Event notifications. A new object created event can send a message straight to SQS, queueing the file for processing without writing any code.

Parsing and processing

4. SQS

The queue decouples the two halves of the service and adds flexibility and resilience:

• Automatic retries. If processing fails, the message reappears on the queue once its visibility timeout expires. Processing calls an external API that could have problems, and n retries cover random failures outside our control.
• Dead-letter queue (DLQ). After a set number of failed attempts, the message moves to a DLQ. That tells us which file needs inspecting, whether the problem is our code or the file, and stops a bad file wasting time and money or blocking the others.
• Elasticity. One Lambda is enough to process files today. If volumes grow, the event source mapping scales out more Lambdas based on queue depth.

5. Lambda (file processing)

This Lambda reads each file, transforms the data, calls a third-party service to validate some attributes and stores the result in a DynamoDB table for other services to use.

• An S3 gateway VPC endpoint keeps the file stream inside the AWS network, off the public internet, and avoids S3 data transfer costs.
• Batching SQS messages would save invocation costs, but it makes a single failed file harder to debug, and the requirements don't need it.
• An explicit iamRole grants access to the one DynamoDB table by its resource ID, which makes least privilege easy.
• Running the Lambda in the VPC means a security group can restrict which external services it reaches.

6. SNS

SNS sends a notification after each file is processed. That could be an email, or a log of success. Once there's a consistent baseline, an alarm can trigger a manual review when the count drops below it.

Worth a mention

I've left out configuration this scenario doesn't need, to keep the post short. A few options are worth considering for similar systems:

• FIFO queues. Guarantee order and prevent duplicate messages, which matters if duplicates can't happen or order is crucial.
• S3 cross-region replication. For a mission-critical system, or files that can't be recreated, replication belongs in the disaster recovery plan.
• Batch size. More messages per Lambda invocation makes individual files harder to debug, but improves throughput with fewer cold starts and saves some cost.
• AWS Transfer Family. Slightly out of scope, but if the FTP server ran on AWS Transfer Family, it would scale automatically and offer workflows that transfer files straight to S3 or EFS. Here, that would save work and remove complexity.

The result covers what's expected of a standard file processing system: one bad file doesn't stop the rest, and the culprit is flagged for inspection. Much of the finer detail is skipped to keep it short.