There's no way to use a X.509 certificate/key for S/MIME from a CA which technically isn't trusted. Neither if it's your certificate (for sending / signing the mail) nor if it's the receivers certificate (for encrypting the mail.

Maybe you don't want to trust a whole CA. But you may know that a certain certificate is trustworthy (e.g. by comparing the fingerprint or because you created the private key and anyone else has a copy).
Sadly there's no way to technically mark a single X.509 / S/MIME certificate trustworthy. That's only possible for CAs.
(maybe there should be such a possibility for single X.509 certificates - something to think about for Kleopatra - but as far as I know that's not the way X.509 works)
(yes I know, I should use PGP for that trust model, but my contacts sadly don't share that view and I don't want to trust their whole stupid CA - nevertheless I need to encrypt my emails to them)


If sending an email, using an technically untrusted certificate for yourself, KMail will just say (a situation that may also occur if someone else needs you to use a certificate from a CA you don't like): "Could not compose message: Not trusted"
No further explanation what's not trusted.
Instead there should be a warning, that you own key isn't trusted. And there should also be the possibility to say "send anyway". Because, as said, you know that you can trust a single key (but you can't technically set that mark to a single X.509 key), but you don't want to trust the whole CA.


Similar thing the other way around:
Send a message to a receiver who's key technically isn't trusted. KMail will give you a short warning, saying:
|  One or more of the OpenPGP encryption keys or S/MIME
|  certificates for recipient "recipient@example.com" is not
|  fully trusted for encryption.
You can click "Cancel" or "Continue" and you can also select "Do not ask again".
But also if you choose "Continue" KMail will refuse to send the mail, telling you: "Could not compose message: Not trusted"

Reproducible: Always