A magic system is the set of rules that govern magic in a story: what it can do, who can use it, where its power comes from, and what it costs or cannot do.
Where the term comes from
A guest article on Jane Friedman's site defines a magic system as a structured set of rules and principles of magical powers in a work of fiction, and treats it as one element of worldbuilding, like geography, historical era and social customs. Its example is a Charms class in the Harry Potter books, where a joke only works because the author has already established that the spell must be pronounced correctly. The most cited craft statements on the subject are Brandon Sanderson's laws. The first law says that an author's ability to solve conflict with magic is directly proportional to how well the reader understands it. The second law says limitations matter more than powers, and uses Superman to argue that what makes him interesting is his code of ethics and his weakness to kryptonite, not his strength. The third law says to expand what you already have before adding something new.
An example
In the city of Marl, scribes can bind a written promise so that it holds in the world, as firmly as a wall. The cost is memory: each binding takes the scribe's recollection of the day on which it was written. The limits are specific. A binding cannot be made about a future the signer does not control, and it breaks if one party is deceived about its terms. When Tamsin Hollis, a young scribe, needs to stop a dockside riot, the reader already knows she cannot simply bind the crowd. She has to trick both sides into agreeing to terms, and she has to choose which day of her life to lose.
Common mistakes
- Powers with no limits or costs, so every problem can be solved by the same move.
- Breaking a rule when the plot needs it. This is the situation Sanderson's first law describes: a solution the reader never had the chance to understand.
- Explaining the full system in one long passage instead of letting it appear through use.
- Setting a cost that no one ever pays.
How to check your manuscript
Write the rules of your system in a few short sentences: what it can do, what it cannot, what it costs. Then list every time magic appears in the book and mark whether it follows those rules. For each conflict solved with magic, check that the reader understood the relevant rule before that scene. Finally, ask what each limit does for the story: a good one creates a choice, a risk or a sacrifice. The guide to creating a magic system walks through the decisions in order, and the entry on deus ex machina describes what happens when the rules fail.
In Musael, continuity checks compare the rules you set with the manuscript and show the sentence that breaks one and the scene it came from.