Solaris let me in with different password with the same 8 first characters

The operating system stores a hash of the password in /etc/shadow (or, historically, /etc/passwd; or a different location on some other Unix variants). Historically, the first widespread password hash was a DES-based scheme which had the limitation that it only took into account the first 8 characters of the password. In addition, a password hashing algorithm needs to be slow; the DES-based scheme was somewhat slow when it was invented but is insufficient by today's standards.

Since then, better algorithms have been devised. But Solaris 10 defaults to the historical DES-based scheme. Solaris 11 defaults to an algorithm based on iterated SHA-256 which is up to modern standards.

Unless you need historical compatibility with ancient systems, switch to the iterated SHA-256 scheme. Edit the file /etc/security/policy.conf and change the CRYPT_DEFAULT setting to 5 which stands for crypt_sha256. You may also want to set CRYPT_ALGORITHMS_ALLOW and CRYPT_ALGORITHMS_DEPRECATE.

Once you've changed the configuration, run passwd to change your password. This will update the password hash with the currently configured scheme.

This is expected, at least on a default Solaris 10 and older configuration.

Your system is using the legacy Unix crypt_unix algorithm which indeed limit the number of characters used to eight.

This is documented in the passwd manual page:

Passwords must be constructed to meet the following requirements:

  Each password must have PASSLENGTH characters, where PASSLENGTH is  defined  in
  /etc/default/passwd  and  is  set  to  6. Setting PASSLENGTH to more than eight
  characters requires configuring policy.conf(4) with an algorithm that  supports
  greater than eight characters.

As this algorithm is essentially obsolete. You should switch to something better (available values listed in the crypt.conf manual page) by setting the CRYPT_ALGORITHMS_DEPRECATE and CRYPT_DEFAULT entries the /etc/security/policy.conf file.

See http://docs.oracle.com/cd/E19253-01/816-4557/concept-63/index.html

See this thread on Oracle forums:

The behavior you describe is expected when using the default "crypt_unix" password encryption scheme. This scheme will only encrypt the first eight characters of a password, and thus only the first eight characters need to match when the password is typed in again. It is not a "bug", but a known limitation of the algorithm - it is largely kept around for backward compatibility, and unfortunately is set as the default on Solaris systems when installed.

To resolve this, set your OS to use MD5 or Blowfish algorithms instead of crypt_unix.

This can be changed in the /etc/security/policy.conf file. You can set crypt algorithms to allow, and there is also a setting to deprecate (forbid) the use of the "crypt_unix" algorithm and change the default to a more secure one.

See your "Solaris 10 System Administration Guide: Security Services" for more information.

See also Changing the Password Algorithm (Task Map) and especially How to Specify an Algorithm for Password Encryption:

Specify the identifier for your chosen encryption algorithm.

...

Type the identifier as the value for the CRYPT_DEFAULT variable in the /etc/security/policy.conf file.

...

For more information on configuring the algorithm choices, see the policy.conf(4) man page.