valid bracket list in prolog

your append(Rest, [')'], T) will parse until the end of the list, but it is not said that the opening bracket will eventually match with the last closing bracket, for example ()() does not.

That being said, I think you make things overcomplicated. Instead of obtaining all sorts of sublists, you can use a single scan here: you use an accumulator that you initialize with 0, and the accumulator should eventually end with 0 and never be less than zero, so:

checkbrackets(B) :-
    checkbrackets(B, 0).

checkbrackets([], 0).  %% ← at the end, zero
checkbrackets([')'|T], N) :-
    N > 0,   %% ← always greater than or equal to zero.
    N1 is N-1,
    checkbrackets(T, N1).
checkbrackets(['('|T], N) :-
    N1 is N+1,
    checkbrackets(T, N1).

Is it possible to write such a test without additional arguments like counters?

I'm fairly sure it's not possible to write such a test (edit: that does a single pass through the list) without tracking additional information such as a counter or a stack. This is because the language you are parsing is a proper context-free language as opposed to a regular one. Parsing context-free languages requires some sort of unbounded state representation, while regular languages get away with finite states.

You would typically handle that extra state using arguments. Possibly hidden ones using definite clause grammars (DCGs). But here -- and I very strongly suggest you do not use this for anything -- is a way of storing that state not in an extra argument but at the head of the list itself.

First, make sure we are using useful syntax for parsing:

:- set_prolog_flag(double_quotes, chars).

This means that anything between double quotes will get interpreted as a list of characters, so you can write "(()" equivalently to the very unreadable ['(', '(', ')'].

Here is the code itself:

checkbrackets([]).
checkbrackets(['(' | Xs]) :-
    checkbrackets([count(1) | Xs]).
checkbrackets([count(0)]).
checkbrackets([count(N), '(' | Xs]) :-
    N1 is N + 1,
    checkbrackets([count(N1) | Xs]).
checkbrackets([count(N), ')' | Xs]) :-
    N > 0,
    N1 is N - 1,
    checkbrackets([count(N1) | Xs]).

This "replaces" the first opening parenthesis with a counter initialized to 1. It increments and decrements that counter as it consumes other opening or closing parentheses. At every update of the counter, the new value is pushed to the front of the list that is passed into the recursive call. The predicate succeeds when all parentheses in the list have been consumed and the counter is at exactly 0. (You don't say if you want to accept ()() or not. This implementation resolves this ambiguity in a particular way that might not be what you intended.)

Examples:

?- checkbrackets("").
true.

?- checkbrackets("(()(()))").
true ;
false.

?- checkbrackets("()(()))").
false.

?- checkbrackets("(()(())").
false.

You could use the same trick to parse more complicated languages that need more complex state than a single counter. But you shouldn't. DCGs are the best way to do this in Prolog.

Note that the implementation above does accept a list that is not purely a list of parentheses:

?- checkbrackets([count(0)]).
true ;
false.

It's possible to fix this, but you shouldn't, since you shouldn't use this approach at all.

For the sake of completeness, here is a solution without additional arguments.

checkbrackets([]).
checkbrackets(['('|Rest]):-
    append(Sub1,[')'|Sub2],Rest),
    checkbrackets(Sub1),
    checkbrackets(Sub2).

It simply follows the definition of "properly parenthesized" expression. Either it is empty, or it starts with a (, followed by a properly parenthesized subexpression (Sub1), followed by ), followed by another properly parenthesized subexpression (Sub2).

However, it is fairly inefficient as compared to the direct solution with one extra argument presented by Willem Van Onsem. The main issue is that the append/3 call needs to non-deterministically "guess" the position of the matching closing parenthesis and this generates a lot of backtracking.