These notes are written with a focus on the usage of RSA public-key cryptography in the Asterisk (IAX2) implementation.

Per the IAX2 RFC 5456 in section 8.6.16, the RSA material exchanged in the authentication process follows the official RSA standard used documented in PKCS#1 V2.0 for RSA (RFC 2437). Note that there are later versions of RSA described (now up to V2.2).

IAX2-Specific Notes

During the AUTHREQ process the server sends a random integer (9-digit string) to the client. The client “RSA signs” the SHA1 hash of this integer using its private key and sends back the resulting signature in the AUTHREP message.

The authentication process in the server involves these steps:

Hash the random integer using SHA1.
Decrypt the signature received in the AUTHREP message to get the SHA1 hash that the client received.
Compare the two to ensure a match.

This is an RSVP1 operation as defined in the PKCS#1 terminology.

Notes on PKCS#1 V2.0 (RFC 2437)

An overview of the PKCS#1 standard is here: https://en.wikipedia.org/wiki/PKCS_1. Basically, this standard defines the terminology and data formats that allow the RSA math to be used in a practical application.

Representation of the Public Key

The public key found in a .PEM file is a base-64 encoding of a complicated structure defined in the standard (specified using ASN.1). Part of the structure has two important fields:

The RSA modulus n.
The RSA public exponent e.

The size of the modulus determines the size of the message that will be encrypted.

Encoding the Message: EME-PKCS1-v1_5

“Encoding” here doesn’t mean “encrypting.” The encoding process converts the message bytes (M) into something that can represented as an integer. The most important part is the padding since the encryption wants to work on a fixed block of data.

M is the original message.
k is the length of the modulus (n) in octets.
Message length can be up to k-11 octets. Padding is used to make up for any shortfall.
The encoded message EM ends up being k-1 octets long.
The padding process must add at least 8 octets of padding consisting of pseudorandomly generated nonzero octets. The pad is called PS.
The encoded message will look like this:
```
  EM = 02 || PS || 00 || M
```

Converting an Encoded Message to an Integer: OS2IP (RFC 2437 section 4.2)

The OS2IP operation interprets a byte string as a big-endian representation of an integer. Each byte in the message is treated as a digit in a base-256 numbering system. The first byte in the message becomes the most significant part of the integer.

Encryption Scheme (RSAES-PKCS1-v1_5)

https://crypto.stackexchange.com/questions/3617/how-do-ciphers-change-plaintext-into-numeric-digits-for-computing

Verification (RSAVP1)

A verification primitive recovers the message representative from the signature representative under the control of the corresponding public key. This is exactly the same as decryption.

This requires Modular Exponentiation: https://en.wikipedia.org/wiki/Modular_exponentiation

References

https://www.practicalnetworking.net/series/cryptography/rsa-example/

https://crypto.stackexchange.com/questions/9896/how-does-rsa-signature-verification-work