Yep, pretty much all code for those consoles were written to suit the particular game -- Studios would possibly re-use modified versions of what they had written before if their next game was a sequel, or similar enough -- I've heard of a few times where engines were licensed from other companies who had done similar games (in particular, I'd read that the Southpark N64/PS1/PC game was essentially a mod of Turok, which never appeared on PS1, but did appear on PC), but I don't know how much truth there is to it.

Consoles of that era were so specific and different from one another in ways that go well beyond using different CPUs or GPUs that any common abstraction--like an engine--would be virtually guaranteed to throw away big performance to get there, and none of those consoles had any to spare. cross-platform engines just wouldn't have made any sense.

Even the needs of different games were so pushing these machines to the limit that an engine suitable for multiple kinds of games on just one of the systems would be pretty hard to pull off. That's why what little code reuse there was, was so often limited to the same studio making the same kind of game (or direct sequels).

Truth be told, I'd say half or more of AAA development is still that way -- all in-house, or heavily-modified in-house -- especially for those high-end (often-exclusive) games. Its only been two console generations or so where the idea of a off-the-shelf engine was reasonably viable for AAA titles. Currently, lots of flashy titles get made with those engines too (though, again, often modified in-house) but I bet its not a majority. But for sort of "standard", not-relly-pushing-the-boundaries console games and more casual console games, certainly engines like Unreal, Crytek, Source, and Unity have made serious inroads.

But for sort of "standard", not-relly-pushing-the-boundaries console games and more casual console games, certainly engines like Unreal, Crytek, Source, and Unity have made serious inroads.

I don't disagree -- there's a sort of in-between AAA class of games that's biggish-budget, might be a biggish name from a biggish studio, but don't really push boundaries of performance or design; You do see a good amount of licensing there (Usually unreal, some Unity as you get further from bleeding edge) that's where commercial engines make a really strong showing.

But few of the highest-end or most-unique games get made with commercial engines -- I'm talking your Metal Gear Solids, your Halos, etc -- for whatever reason. I do discount the games that come first-party from the engine vendors from consideration though -- its not really the same thing to say that all of Valves games are made with Source Engine, or Epic's with Unreal engine and those are all very high-end AAA games so engines must dominate; that's not the same kind of endorsement as when outside companies license those engines and make things.

Its certainly become more prevalent and the trend will continue until until it is the majority of games at all levels I'm sure. I don't think we've reached that just yet though. There's the most pressure to adopt commercial engines on cross-platform games, where its most difficult to make something in-house that exceeds commercial offerings across platforms, but I think we tend to expect less of those titles in terms of boundary pushing, you can't avoid compromise when you go cross-platform.

The first "commercial game engine" that comes to my mind that supported multiple platforms and used by several AAA titles that remotely gets close to the modern concept of game engines was RenderWare. It wasn't even a game engine, it was a rendering engine. And it wasn't for PS1 / N64 gen.

Licensing it costed several tens of thousands of dollars AFAIK. Wikipedia has a list of its competitors (Unreal Engine & Frostbite). Doesn't matter who came first, none of them were for the era you're looking for because, like others already explained, it was all handcrafted and kept in house; occasionally having their stuff licensed to other studios.

To be blunt, If you're asking the kinds of questions you are, its likely the case that you're not at all ready to actually tackle programming these machines in the way they need to be programmed.

Even if you're pretty familiar with programming in general, these machines you name are sparsely documented and incredibly complicated.

• The 3DO has a very early ARM processor with an entirely custom floatingpoint unit, and two custom GPUs that split various duties, having to work in tandem. Its got only 2 megabytes of RAM, and 1 MB of VRAM.
• The Jaguar is two custom DSPs strapped to a 68000 that's barely fast enough to coordinate the two -- one runs audio in software, the other runs graphics (mostly) in software. Only 2MB of RAM total.
• The PS1 has a fairly standard MIPS 3K CPU, but a custom Graphics DSP (graphics mostly in software) designed in-house by Ken Kuturagi. Again, 2MB RAM, 1MB VRAM -- and this might be the most approachable system on your list.
• The N64 has a fairly standard MIPS 4K CPU, and a custom graphics chipset designed by SGI, again two custom DSPs -- one runs 3D graphics and audio, the other runs 2D graphics. 4/8MB RAM. In many ways this should be the most friendly on your list, but its probably less understood than the PS1.
• The Saturn had 8 processors of various kinds, drew quads instead of triangles, and hardly anyone but Sega's own arcade developers could make it sing.

What documentation exists is unofficial, mostly reverse-engineered or gleaned from 1000s of pages of technical manuals for the processors that aren't fully custom. If you've never cracked open the datasheet for an obscure RISC CPU and actually understood the words in front of you -- much less its errata -- this isn't the fight you're looking for.

If you're really hellbent on a console, I'd suggest that the Gameboy advance is going to be your best bet -- its simple, well documented, and is modern enough to do interesting things while still providing a good intellectual challenge. The original gameboy or gameboy color is also a good choice, though more limited and more challenging because of it. The Dreamcast isn't half-bad -- probably the most approachable of the home consoles ever, though the Gamecube/Wii was starting to gather a good scene last I followed this stuff.

Of course the kicker is that you're pining to go through all this work and torture and practically no one will have the means, desire, or will to run the fruits of your labor. If you want, you know, customers, you really should just stick to PC or Mobile, or maybe try to get in on indie development for one of the current consoles, though again, the kinds of questions you're asking suggests to me you've got a lot to learn yet before you're ready for anything I've put on the table here.

And to be clear, I am totally *NOT* discouraging you from learning to write games, I'm just saying you should slow down. I suspect you're quite young -- So was I when I first started programming, like many people here. I was probably 10 or 11 when I copied my first BASIC program out of a book my school library had. I spent the summer between 6th and 7th grades, and my 7th grade studyhalls writing text adventures in a notebook because I didn't have my own computer. But I kept reading and got myself an old computer spent the rest of highschool writing increasingly complex games and tools -- I also tried (and mostly failed) to teach myself C (the language was less of a problem, setting up my environment is what killed me) and dabbled even in some gameboy programming.

There's a path for you here, but you'll only be discouraged if you try to skip right to the end.

I saw in your other thread that you were considering getting a Raspberry Pi -- do that (Get a Pi 3) its a whole computer you can do whatever you want with, and its got tons of programming environments available for it. Or get a used PC or laptop that'll make a serviceable Linux machine (say a dual-core with 4GB of RAM, which you can probably get for not much more than a Pi 3 and which someone might very well give you for free.) a more-standard PC than what a Pi is means better support for software out of the box. Just program -- start small, don't worry about what its going to run on. Program, program, program.