Count repeating integers in an array

Assuming x is sorted, here's one vectorized alternative using unique, diff, and cumsum:

[~, index] = unique(x);
y = ones(size(x));
y(index(2:end)) = y(index(2:end))-diff(index).';
y = cumsum(y);

And now you can apply your final sorting:

>> [~, ind] = sort(y);
>> x_relative_sort = x(ind)

x_relative_sort =

     1     2     3     4     6     1     2     4     6     1     2     6     1     6     1

If you have positive integers you can use sparse matrix:

[y ,~] = find(sort(sparse(1:numel(x), x, true), 1, 'descend'));

Likewise x_relative_sort can directly be computed:

[x_relative_sort ,~] = find(sort(sparse(x ,1:numel(x),true), 2, 'descend'));

Just for variety, here's a solution based on accumarray. It works for x sorted and containing positive integers, as in the question:

y = cell2mat(accumarray(x(:), x(:), [], @(t){1:numel(t)}).');