Is it possible to change the log level of an AWS Lambda at runtime?

Yes it's possible! You need to take care of two things:

• One, update your lambda environment variable
• Two, ensure that log4j will pick up the update

For the first problem, all you need to do is pass a lambda environment variable with name JAVA_TOOL_OPTIONS and value "-DLOG_LEVEL=DEBUG".

For the second point, you can add this to your Java project src/main/resources/log4j.properties with content something like

log4j.rootCategory=${LOG_LEVEL}, LAMBDA
LOG_LEVEL_PATTERN=%5p
LOG_PATTERN=[%d{yyyy-MM-dd HH:mm:ss.SSS}] boot%X{context} ${LOG_LEVEL_PATTERN} [%t] - %c{1}: %m%n
log4j.appender.LAMBDA=com.amazonaws.services.lambda.runtime.log4j.LambdaAppender
log4j.appender.LAMBDA.layout=org.apache.log4j.PatternLayout
log4j.appender.LAMBDA.layout.conversionPattern=${LOG_PATTERN}

And that's it!

If everything goes well you should soon see in your logs a line reading something like

Picked up JAVA_TOOL_OPTIONS: -DLOG_LEVEL=DEBUG

and hopefully start seeing some debug statements.

Notice how LOG_LEVEL is embedded in the value of the lambda variable as opposed to being the variable name. This is a useful indirect way to feed JVM arguments down a lambda which you can then use as system properties. Kudos to https://zenidas.wordpress.com/recipes/system-properties-for-a-java-lambda-function/

I was able to resolve this by using the amazon version of log4j2, and making a change to the log4j2.xml configuration file Add these dependencies to maven

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-lambda-java-log4j2</artifactId>
    <version>1.1.0</version>
</dependency>
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>2.11.2</version>
</dependency>
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-api</artifactId>
    <version>2.11.2</version>
</dependency>

use the environment variable in the level of the logger in the configuration

<?xml version="1.0" encoding="UTF-8"?>
<Configuration
    packages="com.amazonaws.services.lambda.runtime.log4j2">
    <Appenders>
        <Lambda name="Lambda">
            <PatternLayout>
                <pattern>%d{yyyy-MM-dd HH:mm:ss} %X{AWSRequestId} %-5p %c{1}:%L - %m%n</pattern>
            </PatternLayout>
        </Lambda>
    </Appenders>
    <Loggers>
        <Root level="${env:LOG_LEVEL}">
            <AppenderRef ref="Lambda" />
        </Root>
    </Loggers>
</Configuration>

finally, use the log4j2 logger in the lambda itself

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;

public class MyLambdaHandler implements RequestHandler<Void, Void> {

    private static final Logger LOGGER = LogManager.getLogger(MyLambdaHandler .class);

    public Void handleRequest(Void myVoid, Context context) {

        LOGGER.error("Log info enabled: {}", LOGGER.isInfoEnabled());
        LOGGER.info("Info messge");
        LOGGER.error("Log error enabled: {}",  LOGGER.isErrorEnabled());
        LOGGER.error("Error Message");
        LOGGER.error("Log trace enabled: {}",  LOGGER.isTraceEnabled());
        LOGGER.trace("trace message");
        LOGGER.error("Log warning enabled: {}",  LOGGER.isWarnEnabled());
        LOGGER.warn("warn message");
        LOGGER.error("Log debug enabled: {}",  LOGGER.isDebugEnabled());
        LOGGER.debug("debug message");
        return null;

    }

}    

then set the LOG_LEVEL environment variable to the appropriate level to see the the relevant entries in the logs