How to correctly return data and errors from REST annotated methods?

Having written a Salesforce managed package app making use of the automatic serialization and deserialization based on the method signature, in future I will avoid that pattern. The primary reason is that it is then impossible to change the format of the data (because of managed package version compatibility constraints) as the OP mentions, but it also limits the error handling choices. (And the return of JSON_PARSER_ERROR errors rather than an opaque error for invalid JSON requests also raised some security review flags.)

I recommend this approach (providing you are happy to stick with JSON):

@RestResource(urlMapping='/accounts/*')
global with sharing class AccountRestService {
    @HttpGet
    global static void getAccounts() {
        RestResponse res = RestContext.response;
        try {
            List<Account> accounts = [SELECT Id, Name FROM Account];
            res.responseBody = Blob.valueOf(JSON.serialize(accounts));
            res.statusCode = 200;
        } catch (Exception e) {
            res.responseBody = Blob.valueOf(e.getMessage());
            res.statusCode = 500;
        }
    }
}

The pain of a few extra lines of code is more than outweighed by the flexibility to change the requests and responses as needed in the future. And it allows a more RESTful style of error handling.

You can write a Wrapper object instead of the actual object .

Create a wrapper class like below

public class ResponseWrapper{
    public list<Account> lstaccounts;
    public boolean isError ;
    public string errorCode;
    public integer statusCode;
       public ResponseWrapper(){
         lstaccounts = new list<Account>();
         isError= false;
    }

}

Return the wrapper object instead

@RestResource(urlMapping='/accounts/*')
global with sharing class AccountRestService {
@HttpGet
global static ResponseWrapper getAccounts() {
    ResponseWrapper resWrap = new ResponseWrapper();
    List<Account> accounts;

    try {
        accounts = [SELECT Id, Name FROM Account];
        res.lstaccounts = accounts;
    } catch (Exception e) {
        RestResponse res = RestContext.response;
        resWrap.isError = true;
        res.responseBody = Blob.valueOf(e.getMessage()) = resWrap.errorCode;
        res.statusCode = 500 = resWrap.statusCode;
    }

    return resWrap;
  }
}