This will of course make the games a little more expensive, since that would lead to more parties making a profit off of the games (Nintendo, stores, and third party developers).
One potential problem is that up until the N64 third party developers had to buy the cartridges directly from Nintendo if they wanted to sell their game - something they just had to suck up until the Sony Playstation hit. Given the option of having 10x the memory and ridiculously cheaper production third parties migrated to Sony in droves, leaving the N64 with a comparatively tiny, expensive library.
I don't know if Nintendo kept up the practice of selling blank carts to third party GBA/DS/3DS devs (I assume they would've), but if Nintendo decides to go back to/continue with this practice for the NX they might struggle to garner interest from third parties. They've been dominant in the handheld market so publishers will still suck it up to get a game on the 3DS, but for a console release I can't see why ANY developer would choose to pay MORE to develop for Nintendo rather than just developing for Sony/Microsoft.
This is where Nintendo has failed for decades now. They still haven't realised it's not 1989 anymore. Nintendo doesn't dominate gaming, and they can't continue to do as they please and expect third parties to just eat it "because they're Nintendo". I have no problem with them going back to carts in theory, but if they don't make it enticing to developers then no one will go for it and the system will flop. The NX carts have to be as cheap, if not cheaper, to produce than a bluray disc. I don't know if they're at that stage yet - they're smaller and likely use less plastic, but they're also more complicated and will thus likely have a larger failure rate. If the carts can't be produced at an equivalent price Nintendo has to offer to make up that cost somehow, otherwise again, third parties won't bother. They would only intentionally eat those extra costs if Nintendo became market leaders again, and there's no chance in hell that'll happen without third party support.
So come on Nintendo, you stingy bastards. Open up those fat Japanese wallets and throw some devs some of that sweet, sweet yen. It'll pay off big time.
The size of the plastic has nothing to do with it; it's all about how much memory they shoved into it. The NES was supersized for no reason, and actually a downgrade of the original Famicom.
It wasn't supersized "for no reason", it was redesigned to look/sound more like a VCR because following the crash of '83 "video game" was a cancerous word. They figured no one would buy a "video game console", but an "entertainment system" on the other hand could sell. So they took the Famicom, stripped it of its colour to make it look less like a toy and more like a piece of home entertainment, made it a front-loader like a VCR instead of a top-loader like the Atari, made the game cartridges more similar in size to video cassettes (people would even wrongly call them "Nintendo tapes") and called it an "entertainment system". There was no technical reason the NES/games were as large as they were (though I think some of the later games had boards that filled up the entire cartridge, so good they had the extra space. Maybe it WAS foresight?) but it's wrong to say there was "no reason".
Regarding capacity, Nicholas Steel has already said why proprietary carts are generally poorly spec'd compared to generic brands. It has nothing to do with audio fidelity, it has nothing to do with the hardware needing more RAM - a 4GB flash drive today can do everything a 4GB DVD can do, and usually faster too. The only reason current propriety carts (like 3DS carts for example) are "weak" is because there's no NEED for them to be any better (from Nintendo's POV). People will buy them because they have to. They don't need to be bigger or faster because they're not competing with anything.
But again, if the NX is going to try to compete with the PS4/XB1 then the cartridges it uses - whether proprietary or generic - have to be competitive with the standard for those machines (which I think is Bluray, at least 50GB). The 8GB 3DS cart won't do, but a 128GB NX cart should be fine (again, as long as Nintendo makes it just as cheap for third parties to publish on carts).
