How to get nth-child selector to skip hidden divs

I would do this with combination of :nth-of-type selector and small modification of your toggle function.

Basic idea is not to remove those .css--all-photo elements from the DOM but wrap them into <hidden> containers. And unwrap() to restore full set. In this case :nth-of-type will do exactly what you did with :nth-child.

var state = false;

$('.hide-others').click(function () {
    if( !state ) {
      $('.css--all-photo').wrap('<hidden>');
      state = true;
    } else {
      $('hidden').unwrap();
      state = false;
    }
})
hidden { display:none; } 

.board-item--inner {
    height:200px;
    background:tomato;
    text-align:center;
    color:#fff;
    font-size:33px;
    margin-bottom:15px;
    border:2px solid tomato;
}
@media (min-width:768px) and (max-width:991px) {
    div.board-item:nth-of-type(2n+1) .board-item--inner {
        border:2px solid #000;
        background:yellow;
        color:#000;
    }
}
@media (min-width:992px) and (max-width:1199px) {
  div.board-item:nth-of-type(3n+1) .board-item--inner {
    border:2px solid #000;
    background:yellow;
    color:#000;
  }
}
@media (min-width:1200px) {
  div.board-item:nth-of-type(4n+1) .board-item--inner {
    border:2px solid #000;
    background:yellow;
    color:#000;
  } 
}
<link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css" rel="stylesheet"/>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
<div class="container">
    <div class="form-group">
        <button class="btn btn-info hide-others" type="button">Hide others</button>
    </div>
    <div class="row">
        <div class="col-xs-12 col-sm-6 col-md-4 col-lg-3 board-item photos-board-item">
            <div class="board-item--inner">1</div>
        </div>
        <div class="col-xs-12 col-sm-6 col-md-4 col-lg-3 board-item photos-board-item">
            <div class="board-item--inner">2</div>
        </div>
        <div class="col-xs-12 col-sm-6 col-md-4 col-lg-3 board-item photos-board-item css--all-photo">
            <div class="board-item--inner">3</div>
        </div>
        <div class="col-xs-12 col-sm-6 col-md-4 col-lg-3 board-item photos-board-item">
            <div class="board-item--inner">4</div>
        </div>
        <div class="col-xs-12 col-sm-6 col-md-4 col-lg-3 board-item photos-board-item">
            <div class="board-item--inner">5</div>
        </div>
        <div class="col-xs-12 col-sm-6 col-md-4 col-lg-3 board-item photos-board-item css--all-photo">
            <div class="board-item--inner">6</div>
        </div>
        <div class="col-xs-12 col-sm-6 col-md-4 col-lg-3 board-item photos-board-item">
            <div class="board-item--inner">7</div>
        </div>
        <div class="col-xs-12 col-sm-6 col-md-4 col-lg-3 board-item photos-board-item css--all-photo">
            <div class="board-item--inner">8</div>
        </div>
        <div class="col-xs-12 col-sm-6 col-md-4 col-lg-3 board-item photos-board-item">
            <div class="board-item--inner">9</div>
        </div>
        <div class="col-xs-12 col-sm-6 col-md-4 col-lg-3 board-item photos-board-item">
            <div class="board-item--inner">0</div>
        </div>
        <div class="col-xs-12 col-sm-6 col-md-4 col-lg-3 board-item photos-board-item">
            <div class="board-item--inner">10</div>
        </div>
    </div>
    <div>

When the user clicks on a button, I hide few blocks by display:none, and the problem occurs. The nth-child selector also counts hidden elements.

Is there way to ignore those specific blocks, so that again every row has different style?

The problem is that the nth-child() selector looks at all siblings under the same parent regardless of styling. It doesn't matter that you've applied display: none because CSS doesn't remove the element from the DOM, and therefore it's still a sibling.

From the spec:

6.6.5.2. :nth-child() pseudo-class

The :nth-child(an+b) pseudo-class notation represents an element that has an+b-1 siblings before it in the document tree, for any positive integer or zero value of n, and has a parent element. (emphasis mine)

In order for the nth-child rules you've declared to work after a user clicks to hide divs, you need to remove the hidden divs from the DOM, so they no longer exist as siblings.

In your question you request a CSS-only solution. But in your comments you say that the HTML is open for changes. You also use a bit of jQuery to hide elements.

With one small line of code added to your jQuery the problem can be solved:

$('.hidden').remove();

The .remove() method takes elements (and its descendants) out of the DOM. In this case it removes all elements with a class hidden.


CORRECTION

The problem with remove() is that elements taken from the DOM with this method can't be restored, and this breaks the toggle function.

Fortunately, jQuery offers an alternative: detach().

The .detach() method is the same as .remove(), except that .detach() keeps all jQuery data associated with the removed elements. This method is useful when removed elements are to be reinserted into the DOM at a later time.

So if we replace the original code...

$('.hide-others').click(function () {
    $('.css--all-photo').toggleClass('hidden');
})

...with this code...

var divs;

$('.photos-board-item').each(function(i){
    $(this).data('initial-index', i);
});

$('.hide-others').on('click', function () {
    if(divs) {
        $(divs).appendTo('.row').each(function(){
            var oldIndex = $(this).data('initial-index');
            $('.photos-board-item').eq(oldIndex).before(this);
        });
        divs = null;
    } else {
        divs = $('.css--all-photo').detach();
    }
});

... the grid works as intended. (code credit: @JosephMarikle)

DEMO

Now, regardless of which divs or how many are hidden, they can be toggled on and off without disrupting the visual design because the nth-child selector is counting only "visible" siblings. No changes to the CSS. No changes to the HTML.