It’s a while since I did much by way of proving of theorems. But I can speak from when I was doing maths research in the 1980s through to 1990s.

I think the main difference between maths theorem proving and most problem solving is the length of time it can take. If you are doing a chess problem or crossword or some such (I’m not actually good at either of those but just as a “for instance”), you expect to finish it in minutes or hours. You wouldn’t expect it to take weeks or months.

Sometimes you can see an answer in a flash, but often, you may spend weeks, even months, trying to find an angle on a way to prove a central difficult theorem. Not that you spend all that time just working on that theorem, well you might, you might spend years on one theorem, some mathematicians do. But more often - it’s like, you have a fair number of things you want to prove, but there’s one central result that’s your main pre-occupation. “If only I could prove that!” So you have it on your mind and you keep niggling away at it.

Sometimes when you make the breakthrough, it’s through dreams. I didn’t actually solve problems in dreams myself, as far as I know, but often when I went to bed with a difficult maths puzzle on my mind, as I woke up the next day, I’d see the solution. It would just fall together and it’s like “why didn’t I see that before?”.

I think myself that it’s actually the transition between sleep and waking, when everything is a bit kind of blurry and you are not sure if you are asleep or awake, that one’s mind is in a way particularly fluid. When that happens, you may find a new approach that somehow gets you out of the logjam or the fixed tracks you got stuck on.

So, how do you start? Well, you aren’t going to suddenly solve it, if you haven’t got it on your mind. You try the most straightforward way of proving it first. Sometimes it is just obvious how to tackle it, and then you just follow your nose as it were, and you get to the answer. But often it isn’t. You try to prove it, and you find a big gap in your proof. So you learn something from that. That way doesn’t work. What else can you try?

You break it up into smaller bits. You think “If I knew xxx was true the theorem would be easy to prove - so can I prove xxx?”. You start with the things you know already and just play around, looking for consequences.

If you are doing research level mathematics then you have probably read many proofs in the past. So you also have various ideas you can try from those proofs “So and so used such and such to solve a problem like this, I wonder if it will work here too?”.

A lot of the research consists of taking results that someone else has already proved - and tweaking their proofs, taking them in unexpected directions.

A lot of it does consist of breaking big problems down into smaller ones, which you then can solve one at a time.

You draw diagrams, you may also try concrete examples. I want to prove something for all numbers - well - how does it work for 2, or 3, or 55 or whatever? Surprisingly often mathematicians don’t bother trying to look at concrete examples of what they want to prove. But sometimes, just by working through it like that, you may spot something about the problem that you just didn’t notice until you tried getting your hands dirty as it were, a bit like working on a bicycle or a car engine. You can’t solve it just by looking at it, you need to maybe take it apart and really get involved in it.

So you do all that and more. But often the solution just comes “out of the blue” as you are waking up, or when you are walking somewhere, or just in the middle of something else, That’s one of the most satisfying things about proving theorems in maths, the way that it can come together like that, suddenly sometimes, it was so confusing and complicated and then, all of a sudden, you get it.

ADDENDUM

A few other thoughts. First, your supervisor makes a huge difference, if you are a post grad in maths. In my case, my supervisor was Robin Gandy (who in turn was a student of Alan Turing). He had been involved in research into the logic of maths (“mathematical logic”) for decades, completing his thesis in 1952, only . Not only that, he’d talked to all the logicians of his time, been to their seminars, and he understood just about everything on the topic - modern maths is so specialized that often even the professors will only have a thorough understanding of a small part of their field of interest. He had a good grasp of pretty much all the research ever done in mathematical logic. He also had an amazing memory for detail, could mention details from a talk someone had given years earlier.

So, when you were stuck, he’d say “have you seen the research by so and so, it might help” and off you’d go and find that yes, what they did was relevant to your problem.

Your colleagues also. My supervisor had two other students at the time, all working in the same general topic area, and we often would bounce ideas off each other. Also the weekly seminars - they weren’t so helpful in a direct way - unless the speaker was involved in a research interest very close to your own - as most of the time you couldn’t understand 90% or even 99% of what the visiting speaker was talking about (unless you were Robin Gandy) but it still helped just to be exposed all the time to these various ideas, to chat to other logicians and so on.