Traditional publishing
In traditional publishing, the process varies between two categories.
For fiction and creative nonfiction (memoir, biography, etc.), you start with a completed manuscript. For other nonfiction categories, you start with a book proposal, with the promise to write the book after the proposal is accepted.
In either case, you need to be represented by a literary agent. Finding an agent to represent you can be a daunting process. There are good reasons for this, no matter how good your book is. It can take months or years to find an agent. Once you have one, and once they find a publisher interested in working with you, it typically takes between one and two years for the book to hit store shelves. This is normal.
In addition, the author’s royalties are, in round numbers, half of the margin on the book. The publisher gets the other half. The upside of commercial publishing is that the publisher pays the cost of production and generally gives the author an advance against future royalties that you don’t have to pay back if the book doesn’t sell (you have to pay them back if you never deliver a manuscript). The commercial publisher has access to bookstores and other big markets that you as an individual self-publisher do not have access to. So if your book is going to sell in the tens of thousands in bookstores, you want to be traditionally published.
Self-publishing
In self-publishing, you take on all the roles and responsibilities of the publisher. You put money at risk, and you get to keep all of the profits in return.
The upside of self-publishing is that you get the book out much faster, typically within six months from when you have a finished manuscript. The downside is that you don’t have access to the kinds of retailers a large publisher has access to. But this is not necessarily a problem. For a book aimed at a niche audience, a bookstore is the worst possible place to sell books. You will be much more successful using social media and personal networks.
You can also be much more nimble in responding to reader feedback than a trade publisher can be. And as the publisher, you have complete control over the quality of the book.
Which pays better?
Don’t quit your day job. A tiny percentage of authors make a living writing books, whether traditionally published or self-published. Write the book because you’re driven to write it, not because it’s going to turn a profit. Few books earn a significant profit, no matter who publishes them.