Here is a sample mail client registration - To register with Default Programs / modern Default Apps. You can modify it accordingly, leaving out non-essential parts.
Windows Registry Editor Version 5.00
;RegisteredApplications
;----------------------
[HKEY_LOCAL_MACHINE\SOFTWARE\RegisteredApplications]
"MyMail"="Software\\Clients\\Mail\\MyMail\\Capabilities"
;Clients Key (The path mentioned for MyMail in RegisterdApplications key)
;------------------------------------------------------------------------
[HKEY_LOCAL_MACHINE\SOFTWARE\Clients\Mail\MyMail]
@="MyMail"
[HKEY_LOCAL_MACHINE\SOFTWARE\Clients\Mail\MyMail\Capabilities]
"ApplicationDescription"="Superfast, Light-weight Mail Client for Windows"
[HKEY_LOCAL_MACHINE\SOFTWARE\Clients\Mail\MyMail\Capabilities\FileAssociations]
".eml"="MyMail.eml"
".nws"="MyMail.nws"
[HKEY_LOCAL_MACHINE\SOFTWARE\Clients\Mail\MyMail\Capabilities\StartMenu]
"Mail"="MyMail"
[HKEY_LOCAL_MACHINE\SOFTWARE\Clients\Mail\MyMail\Capabilities\UrlAssociations]
"mailto"="MyMail.mailto"
;Then create handlers for MyMail.eml, MyMail.nws, MyMail.mailto as referenced above
;EML File Type Handler
;---------------------
[HKEY_CLASSES_ROOT\MyMail.eml]
@="MyMail EML Handler"
[HKEY_CLASSES_ROOT\MyMail.eml\shell\open\command]
@="\"C:\\Program Files\\Windows Mail\\WinMail.exe\" -eml \"%1\""
;NWS File Type Handler
;---------------------
[HKEY_CLASSES_ROOT\MyMail.nws]
@="MyMail NEWS Handler"
[HKEY_CLASSES_ROOT\MyMail.nws\shell\open\command]
@="\"C:\\Program Files\\Windows Mail\\WinMail.exe\" -news \"%1\""
;MAILTO Protocol Handler
;---------------------
[HKEY_CLASSES_ROOT\MyMail.mailto]
@="MyMail MAILTO Handler"
[HKEY_CLASSES_ROOT\MyMail.mailto\shell\open\command]
@="\"C:\\Program Files\\Windows Mail\\WinMail.exe\" -mailto \"%1\""
I've uploaded the same in this Pastebin link.
For official documentation, refer Default Programs Registration at MSDN. Although the docs state it's not valid for Windows 10, it certainly applies (and works perfectly) in Windows 10. The registration part hasn't changed a bit in Windows 10.
Also, see:
Windows 10 program default settings - Microsoft Community