What does "release sequence" mean?

i stumbled over the exact same question like you did. i thought i got the understanding right and then he comes in with this example and only uses std::memory_order_aquire. it was difficult to find any good information on this, but finally i found some helpful sources. the main information i was not aware of was the simple fact, that read-modify-write operations ALWAYS work on the newest/latest value, no matter what memory order given (even std::memory_order_relaxed). this ensures, that you wont have the same index two times in the example. still the ordering of operations can mix up (so you dont know which fetch_sub will happens before the other).

this is an answer of anthony williams himself stating that read-modify-write operations always work on the latest value: Concurrency: Atomic and volatile in C++11 memory model

additionally, someone asked about the fetch_sub in combination with the shared_ptr ref count. here anthony williams responded too and brings clarity into the situation with the reordering of the fetch_sub: https://groups.google.com/a/isocpp.org/forum/#!topic/std-discussion/OHv-oNSuJuk

it means that the initial store is synchronized-with the final load even if the value read by the final load isn't directly the same value stored at beginning, but it is the value modified by one of the atomic instruction which could race into. A simpler example, assuming there are three threads racing which executes these instruction (assume x initialized to 0 before the race)

// Thread 1:
A;
x.store(2, memory_order_release);

// Thread 2:
B;
int n = x.fetch_add(1, memory_order_relaxed);
C;

// Thread 3:
int m = x.load(memory_order_acquire);
D;

What are the possible values read for n and m according to possible results of the race? And what are the guarantees that we have on the ordering of instructions A, B, C, and D based on what we read on m and n? For n we have two cases, either 0 or 2. For m we could read 0, 1, 2, and 3. There are six valid combinations of the two. Let's see each case:

• m = 0, n = 0. We don't have any synchronizes-with relationship, thus we can't infer any happens-before relationship except for the obvious B happens-before C

• m = 0, n = 2. Even though the fetch_add operation read the value written by the store, since the fetch_add has a relaxed memory ordering there is no synchronizes-with relationship between the two instruction. We can't say that A happens-before C

• m = 1, n = 0. Similarly as before, since fetch_add don't have a release semantic we can't infer a synchronizes-with relationship between the fetch_add and the load operation, hence we don't know whether B happens-before D

• m = 2, n = 0. The value we read with the acquire semantic load has been written with a release semantic store. We are guaranteed that the store synchronizes-with the load, hence A happens-before D

• m = 2, n = 2. Same as above, the store synchronizes-with the load, hence A happens-before D. As usual, the fact that the value read from fetch_add is the same as the one stored from thread 1 do not imply any synchronization relationship.

• m = 3, n = 2. In this case the data read by the load has been written by the fetch_add, and the data read by the fetch_add has been written by the store. However because fetch_add has relaxed semantic, no synchronization can be assumed between store and fetch_add and between fetch_add and load. Apparently, in this case no synchronization can be assumed, same as the case m = 0, n = 0. Here is where the release sequence concept comes in handy: the release semantic store in thread 1 will synchronize-with the acquire semantic load in thread 3 as long as the value that is being read has been written in the release sequence, which includes

  1. all the stores performed later in the same thread as the release operation
  2. all the atomic read-modify-write operation which read a value from the same release sequence.

  In this case since fetch_add is an atomic read-modify-write operation we know that the store in thread 1 synchronizes-with the load in thread 3, and thus A happens-before D. We still can't say anything about the ordering of B and C though.

In your case you have this pseoudocode, assuming number_of_items = 2:

// Thread 1
Item[0] = ...;
Item[1] = ...;
count.store(2,memory_order_release);

// Thread 2
int i2 = 0;
while (i2 = count.fetch_sub(1,memory_order_acquire) <= 0 ) sleep();
auto x2 = Item[i2-1];
process(x2);

// Thread 3
int i3 = 0;
while (i3 = count.fetch_sub(1,memory_order_acquire) <= 0 ) sleep();
auto x3 = Item[i3-1];
process(x3);

Let's assume that the first positive value read into i2 is 2, and thus the first positive value read into i3 is 1. Since the value read from Thread 2 has been written from the store in Thread 1, the store synchronizes-with the load, and we know that Item[1] = ...; from Thread 1 happens-before auto x2 = Item[1]; in Thread 2. However the value 1 read from Thread 3 has been written by Thread 2, with fetch_sub which has no release semantic. The fetch_sub from Thread 2 thus does not synchronizes-with the fetch_sub from Thread 3, however since the fetch_sub from Thread 2 is part of the release chain of the store in Thread 1, the store in Thread 1 also synchronizes-with the fetch_sub in Thread 3, from which we know that Item[0] = ...; happens-before auto x3 = Item[0];

What does he mean? That both threads should see the value of count is 20? But in my output count is sequently decremented in threads.

No he doesn't. All modification to count are atomic, so both reader threads would always see different values for it in the given code.

He's talking about the implications of the release sequence rule, namely that when a given thread performs a release store, other multiple threads that then perform acquire loads of the same location form a release sequence, in which each subsequent acquire load has a happens-before relationship with the storing thread (i.e. the completion of the store happens-before the load). This means that the load operation in the reader thread is a synchronisation point with the writer thread, and all memory operations in the writer prior to the store must complete and be visible in the reader when its corresponding load completes.

He's saying that without this rule, only the first thread would be thus synchronised to the writer. The second thread would therefore have a data race in accessing queue (note: not count, which is protected anyway by atomic access). Theoretically, memory operations on data occurring before the store on count could be seen by reader thread number 2 only after its own load operation on count. The release sequence rule assures that this will not happen.

In summary: the release sequence rules assures multiple threads can synchronise their loads on a single store. The synchronisation in question is that of memory accesses to data other than the actual atomic variable being synchronised on (which is guaranteed to be synchronised anyway due to being atomic).

Note to add here: for the most part these kind of issues are only of concern on CPU architectures that are relaxed about reordering their memory operations. The Intel architecture is not one of them: it is strongly-ordered and has only a few very specific circumstances in which memory operations can ever be reordered. These kind of nuances are mostly only relevant when talking about other architectures, such as ARM and PowerPC.