OpenSSL RSA signature verification: hash and padding?

I am trying to write code to verify some RSA signatures. The signatures were made using the OpenSSL command-line tool, using the equivalent of this command line:

openssl dgst -sha1 -sign private_key_file.pem < binary_data_file > sig

I am trying to use `libtomcrypt` to do the verify:

[To see links please register here]

Here is the calling signature of the RSA verification function in `libtomcrypt`:

int rsa_verify_hash_ex(
const unsigned char *sig, unsigned long siglen, // signature to verify
const unsigned char *hash, unsigned long hashlen, // hash value to check against sig
int padding, // defined constant value, see below
int hash_idx, // identifies which hash algorithm, see below
unsigned long saltlen, // specify salt length, see below
int *stat, // output parameter, returns whether verify succeeded or not
rsa_key *key); // RSA public key to use for verify

This function returns a 0 if it operates without error, otherwise returns an error code. If it operates without error, the `stat` output parameter indicates whether the signature verified.

Most of the arguments seem straightforward: pass in the signature to check, the hash value to use to compare it, and the RSA key to use for the check. `hash_idx` is clear from the example code included with `libtomcrypt`; it is an index into a table of supported hash algorithms, and I can find the correct value to use with this code snippet: `hash_idx = find_hash("sha1")`

But I'm wondering about the `padding` and `saltlen` values. `padding` doesn't worry me too much, as there are only two possible values, and I can just try them both. But what should I pass for `saltlen`?

The OpenSSL documentation for the OpenSSL functions for RSA verify don't show a `saltlen` parameter. The man page for `openssl dgst` (i.e. the result of `man dgst`) does not discuss salt.

So my questions:

* How can I determine the correct salt length to use?
* Does OpenSSL's `dgst` command insert any extra stuff in the input, such as: `(stdin)=`

(I found that `(stdin)=` thing by searching StackOverflow:

[To see links please register here]

)

* `libtomcrypt` also has a function called `pkcs_1_pss_decode()` which is documented to "decode a PSS encoded signature block". Is there any chance that this is the function I need to call?

Thanks for any help you can give me.

EDIT: thanks to the help below, from @Jonathan Ben-Avraham, I was able to get this working today. The answers to my questions are, respectively:

* Use length 0 for the salt, no salt at all.
* No, OpenSSL did not insert anything extra such as `(stdin)=`
* I needed to call `rsa_verify_hash_ex()`, and I needed to specify the `padding` argument as `LTC_LTC_PKCS_1_V1_5`.

One thing to emphasize: Be sure to pass the *hash* instead of the actual data. That threw me off for some time. Here's a snippet that works (but use sha256):

<!-- language: c -->

void
verify_tomcrypt(unsigned char *keyblob, size_t klen,
unsigned char *payload, size_t dlen,
unsigned char *signature, size_t slen)
{
rsa_key key;
int stat;
unsigned long len;
unsigned char digest2[SHA256_DIGEST_LENGTH];

ltc_mp = ltm_desc;
register_hash(&sha256_desc);

/* try reading the key */
if (rsa_import(keyblob, klen, &key) != CRYPT_OK) {
printf("Error reading key\n");
exit(-1);
}

int hash_idx = find_hash("sha256");
if (hash_idx == -1) {
printf("LTC_SHA256 not found...?\n");
exit(-1);
}
len = sizeof(digest2);
if (hash_memory(hash_idx, payload, dlen, digest2, &len) != CRYPT_OK) {
printf("sha256 fails...?\n");
exit(-1);
}

if (rsa_verify_hash_ex(signature, slen, digest2, sizeof(digest2), LTC_LTC_PKCS_1_V1_5, hash_idx, 0, &stat, &key) == CRYPT_OK) {
if (stat == 1)
printf("Tomcrypt: Signature OK!\n");
else
printf("Tomcrypt: Signature NOK?\n");
} else {
printf("Tomcrypt: Signature error\n");
}
}

No salt:

First, generate a binary SHA1 hash of your data:

openssl dgst -sha1 -binary -out hash1 some_data_file

This is an SHA1 hash or digest. There is no salt prependended to the file ``some_data_file``. The ``openssl dgst -sha1`` itself does not add salt. Note that the output file is just a 20 byte SHA1 hash with no salt. If there were salt, the hash would have to include it, probably prepended before the last 20 bytes that hold the SHA1 hash.

Next, sign the SHA1 hash file ``hash1`` with your private key:

openssl pkeyutl -sign -in hash1 -inkey privkey.pem -pkeyopt digest:sha1 -out sig1

Now sign the ``some_data_file`` with ``openssl dgst``:

openssl dgst -sha1 -sign privkey.pem < some_data_file > sig2

Finally, compare the two signatures:

diff sig1 sig2

and you should see that they are the same. This tells us that signing the raw SHA1 hash of a file with no salt is the same as using the ``openssl dgst -sha1 -sign`` command to sign the file, so it must be that the ``openssl dgst -sha1 -sign`` command also did not use any salt when generating its SHA1 hash for ``sig2``.

Note also that you cannot achieve the same result using the deprecated ``rsautl``:

openssl rsautl -sign -in hash1 -inkey privkey.pem -out sig1

instead of ``openssl pkeyutl``, because ``openssl rsautl -sign`` does not do the ASN.1 encoding of DigestInfo as required by RSASSA-PKCS1-v1_5 defined in e.g. RFC3447 section 9.2 step 2. (This is why you need `-pkeyopt digest:` even though `pkeyutl -sign` itself doesn't do any hashing.) See [this SE post][1] for details.


[1]:

[To see links please register here]