Handling register updates in Modbus systems

It sounds like you have a block of registers that must be read or written atomically (i.e. either read/write all of them, or read/write none of them). Traditionally how you deal with this is to have a set of shadow registers; the ISR fills or uses one table, and the UART subroutines use another. The routines which update the data structures lock the working set so that other routines don't get a partial update. Routines which must access the tables check the lock and either "skip their turn" or wait for the lock to be released.

You make these "critical sections" (sections where accessing the table is not allowed) as fast as possible, which is why I recommended a "shadow set" for the UART routines since they're probably the slowest things accessing the tables and in fact may be the only routines which must access the whole set as a single unit. You don't lock the table until you are absolutely sure that you need to, and then you lock it only for the length of time that it takes to copy the data.

Something like this:

if (command_is_good && cmd_is_request) {
    lock_working_table();
    memcpy(serial_copy, working_copy, sizeof(*working_copy);
    unlock_working_table();
    prepare_response(serial_copy);
}

if (command_is_good && cmd_is_update) {
    if (verify_table(serial_copy)) {
        lock_working_table();
        memcpy(working_copy, serial_copy, sizeof(*working_copy));
        unlock_working_table();
        prepare_response();
     }
}

You can get similar results by disabling the ADC interrupt, but I've found over the years that there are generally very, very few times you want to actually screw with interrupt enables, and this is almost never one of them. Locking and better design are almost always the proper way to handle this, which is why I added the comment to @kvegaoro's post. It messes with your timing which may or may not be important, and if you disable the UART ISR you could overflow your FIFO.

If you find that you have a large set of registers that must be updated atomically you may also want to break them up in to smaller atomic groups and limit you request size or bounds.

Edit to address @DaveTweed's comment:

As far as a convention used by ModBus devices... The best thing about ModBus is also the worst thing about ModBus; you are free to do things however you like. The specification is very "loose" in that respect. There is no convention that I'm aware of, but ModBus devices which have such a requirement about blocks of registers would have to implement things similar to how I described the solution above. Almost all trivial devices limit you to a single register for read/write operations, neatly sidestepping the issue. :-)