AWS Step Functions with dynamic steps

Note that you can alway create a State Machine dynamically. (from here)

package com.example;

import static com.amazonaws.services.stepfunctions.builder.StepFunctionBuilder.*;
import com.amazonaws.services.stepfunctions.builder.ErrorCodes;

public class StepFunctionsDemo {

    public static void main(String[] args) {
        final StateMachine stateMachine = stateMachine()
                .comment("A Hello World example of the Amazon States Language using an AWS Lambda Function")
                .startAt("Hello World")
                .state("Hello World", taskState()
                        .resource("arn:aws:lambda:REGION:ACCOUNT_ID:function:FUNCTION_NAME")
                        .transition(end()))
                .build();
        System.out.println(stateMachine.toPrettyJson());
    }
}

If it's all dynamic, you have to get somewhat creative to model the process in Step Functions.

One approach could be to build a state machine with all your tasks in it and a orchestrating decider function. The SM would start with the decider and then execute the correct job based on its output. Each job would in turn call the decider again. If it determines the process is done, the SM would end successfully. This is how it might look:

{
    "StartAt": "Decide next",
    "States": {
        "Decide next": {
            "Type": "Task",
            "Resource":"arn:aws:lambda:::function:decider",
            "ResultPath": "$.nextAction",
            "Next": "Choose action"
        },
        "Choose action": {
            "Type": "Choice",
            "Choices": [
                {
                    "Variable": "$.nextAction",
                    "StringEquals": "DONE",
                    "Next": "Process finished"
                },
                {
                    "Variable": "$.nextAction",
                    "StringEquals": "1",
                    "Next": "Job 1"
                },
                {
                    "Variable": "$.nextAction",
                    "StringEquals": "2",
                    "Next": "Job 2"
                },
                {
                    "Variable": "$.nextAction",
                    "StringEquals": "3",
                    "Next": "Job 3"
                }
            ]
        },
        "Job 1": {
            "Type": "Task",
            "Resource":"arn:aws:lambda:::function:job1",
            "ResultPath": "$.jobResult",
            "Next": "Decide next"
        },
        "Job 2": {
            "Type": "Task",
            "Resource":"arn:aws:lambda:::function:job2",
            "ResultPath": "$.jobResult",
            "Next": "Decide next"
        },
        "Job 3": {
            "Type": "Task",
            "Resource":"arn:aws:lambda:::function:job3",
            "ResultPath": "$.jobResult",
            "Next": "Decide next"
        },
        "Process finished": {
            "Type": "Succeed"
        }
    }
}

You will have to go through the execution log to see the progress through the SM, you won't be able to see it very well visually (all the jobs that ran will be green, which doesn't tell you about their succession).