Should services always return DTOs, or can they also return domain models?

it doesn't feel right when domain model leaves business layer (service layer)

Makes you feel like you are pulling the guts out right? According to Martin Fowler: the Service Layer defines the application's boundery, it encapsulates the domain. In other words it protects the domain.

Sometimes service needs to return data object that wasn't defined in the domain

Can you provide an example of this data object?

If we should strictly stick to DTOs, should they be defined in service layer?

Yes, because the response is part of your service layer. If it is defined "somewhere else" then the service layer needs to reference that "somewhere else", adding a new layer to your lasagna.

is it ok to return domain models all the way to controllers, or should we always use DTOs for communication with service layer?

A DTO is a response/request object, it makes sense if you use it for communication. If you use domain models in your presentation layer (MVC-Controllers/View, WebForms, ConsoleApp), then the presentation layer is tightly coupled to your domain, any changes in the domain requires you to change your controllers.

it seems it wouldn't make much sense to create DTO that is the same as domain model)

This is one of the disadvantage of DTO to new eyes. Right now, you are thinking duplication of code, but as your project expands then it would make much more sense, specially in a team environment where different teams are assigned to different layers.

DTO might add additional complexity to your application, but so are your layers. DTO is an expensive feature of your system, they don't come free.

Why use a DTO

This article provides both advantage and disadvantage of using a DTO, http://guntherpopp.blogspot.com/2010/09/to-dto-or-not-to-dto.html

Summary as follows:

When to Use

• For large projects.
• Project lifetime is 10 years and above.
• Strategic, mission critical application.
• Large teams (more than 5)
• Developers are distributed geographically.
• The domain and presentation are different.
• Reduce overhead data exchanges (the original purpose of DTO)

When not to Use

• Small to mid size project (5 members max)
• Project lifetime is 2 years or so.
• No separate team for GUI, backend, etc.

Arguments Against DTO

• Duplication of code.
• Cost of development time, debugging. (use DTO generation tools http://entitiestodtos.codeplex.com/)
• You must synchronize both models all the time. (personally, I like this because it helps know the ripple effect of the change)
• Cost of developement: Additional mapping is necessary. (use auto mappers like https://github.com/AutoMapper/AutoMapper)
• Why are data transfer objects (DTOs) an anti-pattern?

Arguments With DTO

• Without DTO, the presentation and the domain is tightly coupled. (This is ok for small projects.)
• Interface/API stability
• May provide optimization for the presentation layer by returning a DTO containing only those attributes that are absolutely required. Using linq-projection, you don't have to pull an entire entity.
• To reduce development cost, use code-generating tools

I'm late to this party, but this is such a common, and important, question that I felt compelled to respond.

By "services" do you mean the "Application Layer" described by Evan's in the blue book? I'll assume you do, in which case the answer is that they should not return DTOs. I suggest reading chapter 4 in the blue book, titled "Isolating the Domain".

In that chapter, Evans says the following about the layers:

Partition a complex program into layers. Develop a design within each layer that is cohesive and that depends only on the layers below.

There is good reason for this. If you use the concept of partial order as a measure of software complexity then having a layer depend on a layer above it increases complexity, which decreases maintainability.

Applying this to your question, DTOs are really an adapter that is a concern of the User Interface / Presentation layer. Remember that remote/cross-process communication is exactly the purpose of a DTO (it's worth noting that in that post Fowler also argues against DTOs being part of a service layer, although he isn't necessarily talking DDD language).

If your application layer depends on those DTOs, it is depending on a layer above itself and your complexity increases. I can guarantee that this will increase the difficulty of maintaining your software.

For example, what if your system interfaces with several other systems or client types, each requiring their own DTO? How do you know which DTO a method of your application service should return? How would you even solve that problem if your language of choice doesn't allow overloading a method (service method, in this case) based on return type? And even if you figure out a way, why violate your Application Layer to support a Presentation layer concern?

In practical terms, this is a step down a road that will end in a spaghetti architecture. I've seen this kind of devolution and its results in my own experience.

Where I currently work, services in our Application Layer return domain objects. We don't consider this a problem since the Interface (i.e. UI/Presentation) layer is depending on the Domain layer, which is below it. Also, this dependency is minimized to a "reference only" type of dependency because:

a) the Interface Layer is only able to access these Domain objects as read-only return values obtained by calls to the Application layer

b) methods on services in the Application Layer accept as input only "raw" input (data values) or object parameters (to reduce parameter count where necessary) defined in that layer. Specifically, application services never accept Domain objects as input.

The Interface Layer uses mapping techniques defined within the Interface Layer itself to map from Domain objects to DTOs. Again, this keeps DTOs focused on being adapters that are controlled by the Interface Layer.