metadata API: howto deploy only what specified in package.xml

The Metadata API expects a self-consistent package file set, meaning the package.xml and the metadata source are exactly aligned with one another. Salesforce DX's Metadata API commands aren't doing any special magic; it's zipping up that MDAPI source and package file and deploying it just like Ant or Workbench would. MDAPI is much older than Salesforce DX and in many ways a bit weird by the standards of modern deployment tools.

SFDX, when the force:source:convert command is issued, builds such a consistent MDAPI file set from the current DX project. If you want to deploy only a portion of your overall project, you can edit the package.xml and file tree to match, or break up your DX project into logically separate artifacts and convert them individually to MDAPI-deployable packages.

The issue has to do with the tool you are using, rather than with the metadata api itself.

It is entirely possible to have a package that contains a definition of a custom object that is ignored on deployment if it isn't included in the package.xml file.

I believe the relevant setting in the deploy call is autoUpdatePackage. From the metadata api documentation:

If a file is in the .zip file but not specified in package.xml, specifies whether the file is automatically added to the package. A retrieve() is issued with the updated package.xml that includes the .zip file.

Do not set this argument for deployment to production orgs.

---

A simple deployment package with two new objects defined:

[martin@localhost Test-CreateObject]$ unzip -l package.zip 
Archive:  package.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  07-31-2018 09:49   objects/
      426  07-31-2018 09:49   objects/CreateMe_1__c.object
      426  07-31-2018 09:49   objects/CreateMe_2__c.object
      236  07-31-2018 09:40   package.xml
---------                     -------
     1088                     4 files

The contents of the files, showing the package manifest only lists one object:

[martin@localhost Test-CreateObject]$ cat package.xml objects/*
<?xml version="1.0" encoding="UTF-8"?>
<Package xmlns="http://soap.sforce.com/2006/04/metadata">
    <types>
        <members>CreateMe_1__c</members>
        <name>CustomObject</name>
    </types>
    <version>37.0</version>
</Package>
<?xml version="1.0" encoding="UTF-8"?>
<CustomObject xmlns="http://soap.sforce.com/2006/04/metadata">
    <deploymentStatus>Deployed</deploymentStatus>
    <nameField>
        <label>CreateMe 1</label>
        <trackFeedHistory>false</trackFeedHistory>
        <trackHistory>false</trackHistory>
        <type>Text</type>
    </nameField>
    <label>CreateMe 1</label>
    <sharingModel>Private</sharingModel>
</CustomObject>
<?xml version="1.0" encoding="UTF-8"?>
<CustomObject xmlns="http://soap.sforce.com/2006/04/metadata">
    <deploymentStatus>Deployed</deploymentStatus>
    <nameField>
        <label>CreateMe 2</label>
        <trackFeedHistory>false</trackFeedHistory>
        <trackHistory>false</trackHistory>
        <type>Text</type>
    </nameField>
    <label>CreateMe 2</label>
    <sharingModel>Private</sharingModel>
</CustomObject>

The results of the deployment showing just one object created: