I see the main purpose of the flamethrower is pressure. This pressure is used against the opposing pilots and their engineers to control each engagement.
Each weapon operates efficiently within set ranges and pilots can roughly control the battle with this knowledge. However, other than the Carronades and Lumberjack there are no other dedicated control weapons except the flamethrower. Against pilots, the flamethrower zones them into patterns of behaviour where you can start to dictate each encounter. To put it simply, if your opponent wants to fight at close range, flame them. If your opponent wants to engage at long range, shoot them or flank them and then flame them.
Another consideration is that high level engineers are notoriously effective. They can resist some of the most punishing bombardments and keep all the components on a ship at operating levels without breaking their patterns. But when there is a flamethrower, it demands perfection from the engineers. One slip in the cycle and components will catch more than 3 stacks of fire, leading to time wasted trying to put out the fire, causing more components to be damaged, which will allow key components to break, then more time lost on rebuilding the broken components, and then cycle repeats.
Flamethrowers are needed to control the tactical decisions that a pilot can make and to pressure engineers out of their repair and chem-cycle patterns.
This is a very brief explanation of the utility of the flamethrower and there are many other nuanced details that complement the inclusion of the flamethrower in the game. I agree that as a new player the flamethrower is devastating. I know, because I was at one point just like you and just like you I had to face the threat that the flamethrower proposed and overcome it. But the flamethrower isn't overpowered and can be surmounted with the right tactics and experience.
Keep working at it and you'll find your way like we all did. If you need help, feel free to find me in game or crew for other experienced teams and ask them for help. Almost all of us are happy to help as long as you're happy to ask and listen.