Building Python Layer for AWS Lambda with Docker

python aws docker

The other day, I was building something with AWS Lambda that requires an additional package. Usually, AWS has provided some scientific packages such as scikit-learn and pandas as a Layer that is ready to use. Unfortunately, one of the packages that I need (seaborn) isn’t available yet, so in order to use it in my function, I need to create a Layer manually.

Usually, this is pretty straightforward, all you need to do is to install the package inside a folder and zip it. The command to run it:

pip install -r requirements.txt -t python/

This command will install the package listed in the requirements.txt file into a folder python. Then, you could zip this folder and upload it as an AWS Layer.

However, I happen to be using a MacBook Air with M1 Processor. In some cases, when installing something from a source, the AWS Lambda failed to use it due to incompatibility of the processor type. To handle this, I create a docker container with x86 architecture that will do the installing and zipping for me, and then I upload it to the final file to AWS. So far the result is good, all I need to do is change the requirements.txt to what package I want to install, run the docker, copy the zip to my local machine, and upload it.

The Dockerfile configuration I ended up is:

FROM amd64/python:3.7.17-bookworm

WORKDIR /usr/src/app
COPY requirements.txt ./
RUN apt update
RUN apt install -y zip
RUN mkdir -p ./python
RUN pip install -r requirements.txt -t python/
RUN zip -r python

ENTRYPOINT ["tail", "-f", "/dev/null"]

You can find the whole code and the bash script to run it can be found in my git.