Babymonster looks like is raising. I believe is matter of time for them to be in the top spots. Why are not there yet? I think YG sound doesnt work that well without Teddy. Other than that, they have everything to become big.
About Nmixx, i think have to be mostly with JYP incompetence.
About the girls, the only "problem" is that they are not that pretty, they are very nice ladies for sure, but i believe the other groups on top have prettier idols, and in kpop this aspect matters a lot, but i think is not the big issue.
The bigger problem is JYP. JYP cant make very popular music for girlgroups anymore. When others groups with good quality and good sound began to appear in the scene JYP just couldnt compete on top anymore. Twice start to lose relevance to other groups, ofc they have a big strong fandom so the hit was not that strong. That was not the case with Itzy, once a very popular group, now they just keep losing and losing relevance. JYP really fucked up with ITzy.
Now Nmixx. See, i believe JYP thought the mix concept it was going to hit hard. I mean, you look the MV for 0.0 and you can see they spend alot of money in that MV, not other JYP group have this budget for their debut. No other nmixx MV have this visual quality, JYP artistic direction didnt have the talent to pull a concept like this, they are not SM. Other mistake is the lack of a distintive sound and a strong concept. Yes, they mix different musical genres, but what is their sound, their concept? Dark, elegant, girlcrush, etc? they make 0.0 but later they make Dumb, young stupid and then Cool. Is so different that they feel like a different group sometimes.
The good news, looks like JYP is learning from their mistakes, the last album was really nice. The change in the songs are not that jarring. I believe if they keep that route they can bring more fans. Maybe they never will be on top with the other groups, but i believe they can be more popular. Now, they have to leave aside this thing of Kpop, the choreo thing. They have to act full live, make it a easy choreo and when one member is singing the others dance and they go like that. Their biggest assest is their voices. That with a live band and they will impress more people for sure. Too bad JYP is a damn cheap company.