Fina will also aim to establish an 'open' category at competitions for swimmers whose gender identity is different than their birth sex.
The new policy, which was passed with 71% of the vote from 152 Fina members, was described as "only a first step towards full inclusion" for transgender athletes.
The 34-page policy document says that male-to-female transgender athletes could compete in the women's category - but only "provided they have not experienced any part of male puberty beyond Tanner Stage 2 [which marks the start of physical development], or before age 12, whichever is later".
The decision was made during an extraordinary general congress at the ongoing World Championships in Budapest.
It means that transgender American college swimmer Lia Thomas, who has expressed a desire to compete for a place at the Olympics, would be blocked from participating in the female category.
Earlier Fina members heard a report from a transgender task force made up of leading figures from the world of medicine, law and sport.
"Fina's approach in drafting this policy was comprehensive, science-based and inclusive, and, importantly, Fina's approach emphasised competitive fairness," said Brent Nowicki, the governing body's executive director.
Fina president Husain Al-Musallam said the organisation was trying to "protect the rights of our athletes to compete" but also "protect competitive fairness".
He said: "Fina will always welcome every athlete. The creation of an open category will mean that everybody has the opportunity to compete at an elite level. This has not been done before, so Fina will need to lead the way. I want all athletes to feel included in being able to develop ideas during this process."
Former Great Britain swimmer Sharron Davies, who has argued against transgender participation in women's elite swimming, told BBC Sport she was "really proud of Fina".
"Four years ago, along with 60 other Olympic medallists, I wrote to the IOC and said 'Please just do the science first' and no governing body has done the science until now," she said.
"That is what Fina has done. They've done the science, they've got the right people on board, they've spoken to the athletes, and coaches.
"Swimming is a very inclusive sport, we love everyone to come and swim and be involved. But the cornerstone of sport is that it has to be fair and it has to be fair for both sexes.
Asked whether Fina's policy left trans athletes "in limbo" while they waited for an open category to be created, Davies praised Fina for having conversations around trans inclusion that should have "happened five years ago".
"Sport by definition is exclusionary - we don't have 15-year-old boys racing in the under-12s, we don't have heavyweight boxers in with the bantamweights, the whole reason we have lots of different classes in the Paralympics is so that we can create fair opportunities for everybody," she said.
"So that is the whole point of having classifications in sports and the only people who were going to be losing out were females - they were losing their right to fair sport."
However, 'Athlete Ally' - an LGBT advocacy group which organised a letter of support for Thomas in February, called the new policy "discriminatory, harmful, unscientific and not in line with the 2021 IOC principles".
"The eligibility criteria for the women's category as it is laid out in the policy polices the bodies of all women, and will not be enforceable without seriously violating the privacy and human rights of any athlete looking to compete in the women's category," said Anne Lieberman, the group's director of policy and programmes.
Fina's decision follows a move on Thursday by the UCI, cycling's governing body, to double the period of time before a rider transitioning from male to female can compete in women's races.
The issue in swimming has been catapulted into the spotlight by the experiences of American Thomas.
In March, Thomas became the first known transgender swimmer to win the highest US national college title with victory in the women's 500-yard freestyle.
Thomas swam for the Pennsylvanian men's team for three seasons before starting hormone replacement therapy in spring 2019.
She has since broken records for her university swimming team.
More than 300 college, Team USA and Olympic swimmers signed an open letter in support of Thomas and all transgender and non-binary swimmers, but other athletes and organisations have raised concerns about trans inclusion.
Some of Thomas' team-mates and their parents wrote anonymous letters supporting her right to transition, but added it was unfair for her to compete as a woman.
USA Swimming updated its policy for elite swimmers in February to allow transgender athletes to swim in elite events, alongside criteria that aim to reduce any unfair advantage, including testosterone tests for 36 months before competitions.
Last year, weightlifter Laurel Hubbard from New Zealand became the the first openly transgender athlete to compete at an Olympics in a different sex category to that in which they were born.
Dr Michael Joyner, a physiologist and leading expert in human performance
"Testosterone in male puberty alters the physiological determinants of human performance and explains the sex-based differences in human performance, considered clearly evident by age 12.
"Even if testosterone is suppressed, its performance enhancing effects will be retained."
Dr Adrian Jjuuko, an activist, researcher and lawyer
"The policy emphasises that no athlete is excluded from Fina competition or setting Fina records based on their legal gender, gender identity or gender expression.
"[The proposed open category] should not become a category that adds to the already existing levels of discrimination and marginalisation against these groups.
"I see this policy as only the first step towards full inclusion and support for the participation of transgender and gender-diverse athletes in aquatic sports, and there is a lot more to be done."
Dr Sandra Hunter, an exercise physiologist specialising in sex and age differences in athletic performance
"By 14 years or older, the difference between boys and girls is substantial. That's due to the advantages experienced due to the physiological adaptations in testosterone and the possession of the Y chromosome.
"Some of these physical advantages are structural in origin such as height, limb length, heart size, lung size and they will be retained, even with the suppression or reduction of testosterone that occurs in the transition from male to female."
Summer Sanders, former Olympic and world champion in swimming
"This is not easy. There must be categories - women's, men's and of course a category for trans women and trans men.
"Fair competition is a stronghold and staple of our community - this approach safeguards the integrity of the existing sports process in which millions of girls and women participate annually."
The conversation around the inclusion of transgender women in women's sport has divided opinion both inside and outside the sporting sphere.
Many argue transgender women should not compete in women's sport because of any advantages they may retain - but others argue sport should be more inclusive.
World Athletics president Lord Coe has said the "integrity" and "future" of women's sport would be "very fragile" if sporting organisations get regulations for transgender athletes wrong.
British Prime Minister Boris Johnson said in April he did not believe transgender women should compete in female sporting events.
On Sunday, culture secretary Nadine Dorries told radio station LBC she would "encourage other sports" to follow Fina's lead and that "fairness should always trump inclusion as a principle".
The heart of the debate on whether transgender women athletes should compete in women's sport involves the complex balance of inclusion, sporting fairness and safety - essentially, whether trans women can compete in female categories without giving them an unfair advantage or presenting a threat of injury to competitors.
Trans women have to adhere to a number of rules to compete in specific sports, including in many cases lowering their testosterone levels to a certain amount, for a set period of time, before competing.
There are concerns, however, as highlighted in Fina's decision, that athletes retain an advantage from going through male puberty that is not addressed by lowering testosterone.
-- Courtesy of BBC Sport