Clarifying self-signed certificates vs root certificate authority
You should see the certificate infrastructure (that is to say the PKI, Public Key Infrastructur) as a web of trust.
When you are communicating with multiple persons, which wants to mutually authenticated themselves (and their websites), you first choose a common trusted person. This third person becomes a "Trusted Third Party" and will be called the "Certificate Authority" (CA) for the purpose of certificate management.
The CA will create a master key, also known as root CA key. The private key will be kept secret by the TTP at all cost. The public key will be embedded into a certificate, and this certificate will be signed by the public key of the CA. This means the certificate is self-signed and you can verify the signature with the key inside it.
Let's assume now that you want to access Alice's website, but you want it to really be Alice's website and not another person that claims to be her. Alice is part of you community and therefore contacts the CA you agreed to use. She sends the CA a CSR (Certificate Signing Request) which basically is a public key. The CA do the process of verifying that Alice is Alice (for example, the responsible person meets Alice in person and get the CSR from her hands). When the CA is sure Alice is Alice and the CSR belongs to her, the CA can create a certificate in the name of Alice and sign this certificate with the CA private key.
When you connects on Alice's website, you ask for the certificate. Once you get the certificate, you want to verify it's the good one. You can see in the certificate that it has been issue by a CA. If you have the CA key you can verify the signature. If you trust the CA is doing its job correctly, then you can trust that this certificate is a good one and since the CA trusts it authenticated Alice in the first place, then you can trust that this certificate belongs to Alice.
You can then be sure that encrypting a message with this public key will only be read by Alice who has the corresponding private key. You are also sure that anything singed by this certificate is a signature made by Alice.
If Alice wants to authenticate you in return, then you need to do the same things (get a certificate signed from the CA, since Alice trusts the CA as well she will trust this certificate)
In your case, since it seems you master all the endpoints, you could simply create two self-signed certificates (which are in fact two root CA certificates) and exchange the public parts to the other endpoint. Each endpoint will use their own private key to sign outgoing requests and decrypt incoming messages, and the other endpoint certificate to encrypt outgoing message and verify incoming messages.
If you happen to have a lots of endpoint, it could be worth to create your own CA. You generate a root CA certificate then you sign certificate for each endpoints. At each endpoint you can install the public root CA cert. Then each time a endpoint connects to another, the contacted endpoint can send its certificate and the caller can verify it against the root-CA cert.
The first method has the advantage of being straight forward, but compromising of one certificate implies to exchange it for a new one on all endpoints. The second method has the advantage to be scalable, one endpoint does not have to know how many other endpoints exists since it can validate the certificate on the fly anyway. Moreover there are way to manage key compromising with Certificate Revocation Lists for example.
The two options you mention are almost correct: However, you can (and should) install self-signed certificates without them being Certificate Authority certificates.
The difference between a self signed cert and a CA cert is that a CA certificate is a special self-signed certificate with its "basicConstraints" set to "CA:true" (usually with the critical flag set). Additionally, the CA cert's keyUsage sequence lists "cRLSign" and "keyCertSign" as purposes which should not present in a regular (non-CA) certificate.
Now to your two questions:
You can use it as a framework-level or OS-level certificate to your liking, but do not confuse self-signed certificates with CA certificates (see above). Since the certificate seems do be a special purpose certificate for your application only, I would recommend to use it as framework-level cert. Also, installing a trusted OS-level cert might require administrator privileges which not all communication partners might have.
N/A (since the answer to 1. is yes :) )
On a side note: The reason why the distinction between CA and non-CA self signed certificates is relevant is that you should not allow a self-signed certificate, whose purpose is to validate a communication partner, to act as a certificate authority since this might open up additional security implications (i.e. MITM attacks using said certificate as trusted authority).