It's heavily implied that the fertility crisis is also due to men's infertility.
As women become property and status symbols, we see in Testaments that young and fertile women are paired off with the geriatric High Commanders. Older fathers have worse sp*** quality, have less libido, are likely to have children with ADHD, Autism, contribute to worse maternal health outcomes, etc etc compared to fathers younger than age 35.
If this were really about a population crisis, they would invert the Aunt matches whereby Young Men are paired with the Greens.
I think this goes to show how Gilead's subjugation of women is to uphold power structures that benefit old men, and they use religion as their cover. Having a young wife becomes a commodified reward for those who are loyal to Gilead.
I suspect the hoarding of women in a few elite old men would cause elite overproduction and eventually the vast majority of younger men would realize they've been strung along as labor and Canon fodder.
In essence, the least useful demographic in human reproduction are the old men, and yet they are given the most useful demographic in human reproduction - young women.
No wonder Gilead fell after like 25 years. It really isnt sustainable from a population growth lens.