This all is under the umbrella of reverse engineering or software cracking. This is an old e-magazine (https://www.exploit-db.com/ezines/kr5hou2zh4qtebqk.onion/ARTeam/ARTeam.Ezine.Number2.pdf) showing a rather easy way to add a menu to notepad (Windows XP/7/10) to add functionality. This is in chapter 1. In this case only small modifications are done in the original software (resources and loading a dll), but the changes could have been done without a dll library using code caves or a new section instead.
Reverse engineering is a very large area containing many topics, depending on the software, the methods, the hardware, and even the operating systems. So you need to learn a lot. It starts from crackmes, patching, debugging and reading assembler code. Then it goes to anti debugging techniques, manual unpacking of protections/packers, understanding protocols and file formats, keygenning, custom authentication servers, inline patching, copy protections (disc based and online) hypervisors and drivers and much much more.
“Updated README<span>.</span>md”