Hi. Andi summed things up quite nicely, but I will try to add to it.
There are two wanted features: encryption/decryption and signing/verification for (1) PGP/GPG and (2) S/MIME. While there are many similarities, they are not the same.
S/MIME is CMS/CAdES format built into the email text format, and there are multiple versions. The latest version seems to be 4.0 (see RFC 8551). The first task you want to cross-out is having a library capable of producing and parsing CMS/CAdES in binary format (which is essentially an ASN.1 structure). When you have such a library, the next step is correctly handling email structure. Standards define two different email formats with an S/MIME signature. AFAIR, they are multipart/signed and application/x-pkcs7-signature, and they differ in signature inclusion. The first format attaches the signature next to the message (the email is a multipart) and the latter envelopes the message within the signature so the message is not visible in the raw email dump. So, in the context of verification, message extraction and verification will differ a bit. In the context of signing, both formats are probably good to use - multipart/signed for signed-only email and the other for signed and encrypted email. Next, for producing signed and encrypted emails, a decision needs to be made whether to first encrypt the message and then sign it, or vice versa. I recommend to consult the Security Considerations in RFCs with this (and make sure that the CMS/CAdES library can handle both because in general, you never really know what you’re going to receive from other email clients). If we come back to correctly handling email structure, the next big thing is correctly handling different Content-Type values. For example, you may not always receive one of the two mentioned Content-Types (email transport layer is generally unreliable to support newest technologies), and in the worst case, the signed message could be buried somewhere deeper in the email structure, or there could be multiple such messages, or there could be nested messages. In these cases, extracting the messages and signatures takes a bit of work, but support for them will likely be postponed until a significant number of users request it - a priority is, of course, to support the most prevalent inputs. The same can also be said about many things concerning the CMS/CAdES structure, signing certificates and their verification, but this is a security-sensitive context, so attention should be paid to properly implement the generation/verification routines. The key is having someone who knows the difference between critical and non-critical things. When it comes to producing signatures, the key thing is support for various private key storages. People may use (e.g.) their operating systems’ key stores, their browser’s key store (FireFox) or an external storage device, usually with a USB connector. I think all three options have something in common: you need to be able to extract the signing certificate from them and use their PKCS#11 interface to produce the signature value for you without getting access to the private key. For PKCS#11 capability, I recommend to include another library into the project. Especially support for USB devices will take a bit of work and tinkering. Signature verification also requires a list of trusted root/intermediate certificates. Again, they may come from (e.g.) operating systems’ trust stores, browser’s trust store (FireFox) or an external trust store (one shipped with the application or explicitly provided by the user). Occasionally, people will want to add a root/intermediate certificate into the trust store. AFAIR, there is potentially a conflict during signature verification in that which email address to verify against which. AFAIR, the “From” header may not always contain the address of the person who actually sent the message and attached the signature. Also, the certified email address may be specified in several places within the certificate, or there may be multiple addresses within. Extracting a list of possible sender addresses and a list of possible certified addresses, and doing an intersection may not be a secure way of verification. What many people could appreciate is a nice and round verification summary, especially when the feature is still young and not fully tested, or when verifications fails. If it were me, I would add an option to display two things somewhere beside the signed message’s viewing canvas: the Common Name or Distinguished Name of the certified subject and the issuer. This would serve as a quick visual check for the user. Ok, I think that sums up the key issues to solve when bringing support for S/MIME. I would have loved to work on this if I had the time but unfortunately, I don’t.
Personally, I have no experience with PGP/GPG on a technical level but I presume it is something quite similar. One of the key differences is that, in this case, some users may prefer not to trust public keyrings and build their own trust stores of public keys. The hard part though is making the trust stores secure because they will likely contain the private keys as well. On some operating systems, perhaps the provided facilities could be used (e.g. KeyChain on macOS), but I’m not completely sure.
In any way, both these endeavours will not be easy, and will require commitment, time and effort if the goal is to support the needs of many people and do it properly. Especially testing may be difficult.