All 3 technically mean the same thing, but they have different origins for different uses. The first two are porn labels at best. Shemale is derogatory and has been falling out of use. I haven't seen Newhalf used to be derogatory but it would still be rude to refer to real people as Newhalf in English. Transgender is the acceptable term for talking about real people in English.
While "trans" has replaced "shemale" on most porn film sites since they star real people, most boorus I've seen have taken to using "intersex" as a more appropriate tag in place of "newhalf" or "shemale" since typically you tag what you see on a character (genitals and physical appearance), not what you know about a character (gender).
In real life intersex refers to people literally born with sex characteristics that don't conform to the binary. They will usually be assigned a binary gender despite this. If they identify as a different gender later in life then they will then also be transgender.
- - - - - - - - - - - - - - - - - - - - -
NEWHALF
ニューハーフ (nyuu-haa-fu) is a term coined by the Japanese so they could differentiate between futanari and MtF trans porn. It was then loaned from Japanese into English. It literally refers to sex reassignment surgery giving the person a "new half." I'm not sure if it is used for porn films in Japan, but in my observations it usually only applies to hentai or doujin in English speaking countries.
SHEMALE
It's unclear exactly where the word "shemale" originated, but it eventually became widely used in the porn industry as a category for videos featuring MtF actresses. It's considered derogatory. I haven't seen this used on boorus in a long time, and Pornhub has switched to the correct word "trans," but it still exists on sites like xvideos.
TRANSGENDER
Transgender is the umbrella term for people who have a gender different from the gender that is associated with the sex they were born with. Doesn't matter whether they've had sex reassignment surgery or take hormones. Unlike newhalf or shemale, transgender refers to both FtM and MtF. I think it also can apply to nonbinary genders since those are also different from the gender associated with the sex they were born with but I don't really know enough about trans issues to be sure. Unlike the other terms, it was never designed to be a porn label.
- - - - - - - - - - - - - - - - - - - - -
tl;dr
Newhalf is a Japanese word loaned into English to refer to drawn MtF trans porn.
Shemale is a derogatory term that was once widely used in the porn industry as a category for films starring MtF actresses. It's falling out of use.
Transgender is the umbrella term for people who have a gender different from the one associated with the sex they were born with. It can be for both MtF and FtM. Probably for nonbinary genders as well but I don't know enough to be sure.
- - - - - - - - - - - - - - - - - - - - -
Please correct anything I've said if I'm wrong. I hope I've represented this accurately and respectfully.