What caused Sweden to transform from a highly militaristic society (30 years war through the Napoleonic Wars) to one of staunch neutrality?
First of all, Sweden being neutral did not mean it was not militaristic or even belligerent at several times even after the Napoleonic war. In general, Sweden had more soldiers per capita and invested a larger percentage of its GDP into defense spending than most western countries.
Sweden’s foreign ambitions almost all start with weak neighbors or potential threats.
As the Livonian Confederation was falling apart and faced Russian invasion 1558-1583, the Duchy of Estonia – we are talking the nobility and the burghers of Reval (Tallinn) here, the only ones with a voice in Estonia at the time – swore allegiance to Sweden 1561, which was confirmed by all powers 1570 in the Peace of Stettin.
Sweden’s first real war after the final dissolution of the Kalmar Union was the Nordic Seven Years’ War 1563-1570, which was indecisive, but forced the Danish King to finally abandon his claims on the Swedish throne and thus ending a potential restoration of the Kalmar Union.
Johan III, son of Gustav I (Wasa) married a Polish princess, and their son, Sigismund, became King of both Poland-Lithuania and Sweden. He preferred to host his court in Poland and was a catholic. His uncle, Karl, used fears of continental serfdom, catholicism and counter-reformation, his own position as a very independent Duke with vast estates to revolt against the King and siezing the Swedish throne.
During the Russian times of trouble, Sweden took the opportunity to take Ingria and Kexholm during the Russian war 1610-1617 in order to protect Swedish Finland and take control of all ports and river mouths, thus being able to toll all Russian trade (except the one at Archangelsk, but that was closed during a large part of the year).
Sigismund was still King in Poland-Lithuania and still claimed the Swedish throne – the Poles supporting various candidates for Czar of Russia led Sweden to intervene and de la Gardies campaign, which actually instated Vasilij IV and captured Moscow for him, until de la Gardie was defeated by the Poles. The threat of a Polish-Russian alliance was too severe for Sweden to consider not intervening.
This also led to the Livonian War of 1600-1629, in which Sweden defeated Poland-Lithuania and captured Riga and most of Livonia.
Now Sweden had a strange situation – it had a competent army and a Baltic Empire – but at this time, supporting an army was extremely expensive. Sweden was sparsely populated, and although decently well off by GDP per capita, it was poor compared to most European nations. In order to maintain its Baltic Empire, it needed an army. In order to maintain its army, it needed a war. This situation would last until 1660. Sweden went to war with Spain and the Holy Roman Empire, Poland-Lithuania, Denmark and Russia several times during this era.
To maintain the army, it was placed in foreign land, living mostly off the land and pay was maintained with tolls and French subsidies. As the Habsburgs surrounded France by controlling both the Holy Roman Empire and Spain, it was a sensible policy for France to pay to keep the Holy Roman Empire occupied by fighting a relatively unimportant country to the north.
After the death of King Karl X Gustav in 1660, a regency ruled for his son Karl XI until he reached maturity in 1672. At this time the Swedish army was defeated by Bremen (!) 1665-1666 which severely shocked Europe at the time, since Sweden had been seen as more or less invincible. Corruption in the regency council also led to a worsening state of the army and especially the navy and increasing French influence to the extent that France treated Sweden as a client state at the Peace of Fontainebleau 1679. France signed the peace in Sweden’s name without consulting the Swedish government. It was to Sweden’s advantage, as the enemies of Sweden and France were forced to return Sweden’s German provinces captured during the war, but it made King Karl XI wary of French influence. He had no desire to see Sweden as a client of France.
King Karl XI made two great policy changes – first of all, he changed the foreign policy, aligning with the Anglo-Dutch instead of France and ending the long-standing tense relations with the Holy Roman Empire. Secondly, he created an effective army that did not need a war to exist with indelningsverket (the allotment system). Semi-professional soldiers maintained by peasants (who maintained a soldier instead of paying tax) created a superb army that did not cost the state much to maintain in peacetime.
This army served his son, Karl XII very well during the Great Northern War 1700-1721, but Sweden still lost (against Denmark-Norway, Poland-Lithuania, Saxony, Prussia-Brandenburg, Russia and Hannover) and the Baltic Empire was no more. Still, Sweden retained Finland and a small German bridgehead in form of Vorpommern (Nearer Pommerania).
Swedish politicians during the corrupt so called frihetstiden (freedom era), when the members of the estates parliament were openly bribed by the French and Russian embassies to vote in both foreign and internal matters as those countries wished, did not want to let go of Swedish grand power status, and on French instigation, Sweden entered a disastrous war against Russia 1741-1743 – the plan was to capture Saint Petersburg and aid a French-planned coup d’etat to replace the Austria-friendly regime of Anna Leopoldovna (regent of Czar Ivan IV). Once again France used Sweden as a pawn. The war resulted in a peasant revolt which actually captured Stockholm before being cut down by calvary and grape shot. The same government who had started the war against Russia now considered itself forced to seek Russian aid against its own population. The Russians stationed 30 galleys and 12 000 men north of Stockholm for some time – effectively, Sweden was then a pawn of Russia.
Secondly, Sweden attempted to regain lost parts of Pommerania by entering the Seven Years’ War (the Swedish participation lasted 1757-1762), which was indecisive, even if Sweden completely destroyed the infant Prussian navy in the Battle of Frisches Haff 1759. Again this war came at French instigation.
Gustav III, who had launched a coup d’etat 1772 and faced stiff opposition against his new absolutist rule from the nobility who had lost a lot of power, launched a war against Russia 1788-1790, in the hopes of reducing Russian influence in Sweden and regain territories lost. The war was indecisive, partially because noble officers refused to obey orders and wrote a letter to Czarina Catherine the Great, trying to broker a peace since the Swedish declaration of war technically was illegal. However, the Russian status as a guaranteer of the 1720 Swedish constitution (which had been dissolved by Gustav III in 1772) was ended, which meant that Russia no longer had a formal right to intervene in Swedish politics.
Durign this time, Sweden once again aligned with Francem something which ended with the French revolution. Gustav III had plans to aid the royal family, and the Swedish nobleman Axel von Fersen, who had very close ties with Queen Marie Antoinette (he was probably her lover) attempted to aid their escape.
Gustav IIIs son, Gustav IV Adolf maintained a strict anti-Napoleonic policy. In 1807 a small Swedish army basing out of Pommerania was captured by the French, but were released when they tricked the French into believing they were going to launch a pro-French coup or even a revolution in Sweden. Sweden had to join the continental system formally in the peace, but never did so.
So, 1808 the Russians, on French instigation, attacked Sweden to force it into the continental system. Due to confusion, noble opposition to the King and bad plans, Sweden lost Sveaborg and the whole of Finland quickly. King Gustav IV Adolf was deposed, and the intended successor of his brother, Karl XIII, the Danish-Norwegian Christian August (taking the name Carl August in Sweden) died from a stroke (rumored to have been poisoned, and the earlier mentioned Axel von Fersen was lynched by a mob over it).
Since Russo-French relations were deteriorating, a few Swedes launched a mini-coup and invited the French Marshal Jean-Baptiste Bernadotte to become crown prince and regent of Sweden, thinking he would recapture Finland in alliance with Napoleon. He accepted, and was elected crown prince. However, he saw the writing on the wall and refused to ally with Napoleon and attempt to retake Finland during the French 1812 campaign. Instead Sweden partook in the War of the Sixth Coalition 1812-1814 as one of the anti-French allies and received Norway as a price for its participation at the Congress of Vienna.
King Karl XIV Johan, as Jean-Baptiste Bernadotte became known in Sweden, started the long path of Swedish neutrality. As he had no designs to regain territory lost to Russia, and had traded away Swedish Pommerania for Norway at the Congress of Vienna (Denmark, which got it, traded it to Prussia and got the Duchy of Lauenburg instead). Seeing that Sweden was not a great power, and that designs towards that would only lead to ruin, he devised that “Sweden shall be free of foreign entanglements”. No alliances, and no foreign influence. King Karl XIV Johan and his government and successors and their governments saw that foreign influence had caused much more problems than they had resolved, and decided that neutrality was much better option.
Russia was content with Finland – they had mostly wanted Sveaborg as an outer fortress to protect Saint Petersburg anyway, and with the trade-away of Swedish Pommerania, Prussia (and later Germany) did not have any territorial designs on Sweden either.
While Sweden has been close to going to war several times since – an attempt to build a grand European coalition against Russia (including Austria and Prussia) to re-capture Finland was made during the Crimean War 1853-1856, and Sweden sent troops to aid Denmark directly 1848 and volunteers 1864. However, the Scandinavian Union that some wanted never materialized as Denmark was defeated by Prussia 1864 and lost Lauenburg, Schleswig and Holstein.
Sweden sent volunteers to Finland and Estonia 1917-1918 and to Finland 1939-1940 (declaring itself not neutral but rather non-belligrent in the winter war) and again to Finland 1941-1944, but has not participated in any war except UN-missions such as in Kongo.
Since neutrality has worked very well for Sweden during the 1814-1989 era, there has been no reason to revert this policy.