What an anvil job costs
In Java Edition each anvil job costs the enchantments on the right-hand thing, each one's level times its multiplier, plus a penalty for how many times the left and right things have already been through an anvil. The multiplier comes from the enchantment and is halved when it comes off a book: Protection, Sharpness, Efficiency, Power, Piercing and Loyalty cost 1 a level from a book, the rare ones like Silk Touch, Infinity, Thorns and Channeling cost 4. The penalty doubles with every use (0, 1, 3, 7, 15, 31 levels), and whatever comes out carries one more use than the more-used of its two inputs.
How the order is found
Every way of combining the books is tried. The calculator looks at each group of books and each number of anvil uses the result could have, keeps the cheapest way to make it, and builds bigger groups from smaller ones, so the answer is the true minimum rather than a rule of thumb. It also drops any order with a single job of 40 levels or more, which survival mode refuses as Too Expensive. When two orders tie on levels, it picks the one whose dearest job is smaller, then the one that leaves the item with fewer anvil uses.
Reading the steps
Each step says what goes in the left slot and the right slot, what it costs, and the running total. The item always goes on the left, since the left slot is the thing being upgraded. A step can join two books first; the joined book then shows up later in the right slot. You need at least as many levels as a step costs before you start it, and doing the steps in order never asks for more than the dearest one.
Java Edition numbers
The costs, multipliers and which enchantments fit which item are those of Java Edition 1.21. Bedrock Edition works out some anvil costs differently and allows a few different enchantments on some items, so its totals can differ. Each book is assumed to hold one enchantment at the level you picked; the item's material (iron, diamond, netherite) makes no difference to the cost.
Questions
Why does order matter at all?
Because of the penalty. Every anvil job adds a use to what comes out, and each use doubles the penalty on the next job. Putting cheap books on early and joining books together before they meet the item keeps the uses low, so the same books can cost very different totals depending on the order.
What does Too Expensive mean?
Survival mode refuses any single anvil job that would cost 40 levels or more. If no order keeps every job under that, the calculator says so and shows the cheapest order anyway, with the jobs that are too dear marked, which would only work in creative mode. Leaving out a book or two, or starting from an item with fewer anvil uses, usually brings it back under.
Why can I not pick two of these together?
Some enchantments shut each other out: the four kinds of Protection, Sharpness with Smite and Bane of Arthropods (and Density and Breach on a mace), Fortune with Silk Touch, Infinity with Mending on a bow, Depth Strider with Frost Walker, Multishot with Piercing, and Riptide with Loyalty or Channeling. Once you pick one, the others in its set are greyed out.
My item already has enchantments. Can I use this?
Set how many times it has been through an anvil, which sets its penalty, and pick only the books you still want to add. The calculator assumes the books add new enchantments rather than raising one the item already has.
Is the total in levels or experience points?
Levels, the number the anvil shows. A level near 30 holds far more experience points than one near 5, so it is cheaper in points to do the steps while your level is low and gather more between them.