Nintendo 64 programming characteristics

The programming characteristics of the Nintendo 64 present unique challenges, with distinct potential advantages. As with many other game consoles and other types of embedded systems, the Nintendo 64's architectural optimizations are uniquely acute, yielding weaknesses and thus specific software engineering challenges. First released in 1996, the Nintendo 64's architecture yields programming requirements which The Economist described as "horrendously complex." The weaknesses were said to be caused by a combination of oversight on the part of the hardware designers, limitations on 3D graphics and general computing technology of the time, and manufacturing capabilities. Ultimately, these limitations would be overcome and the advantages would be more thoroughly realized by sufficiently experienced and dedicated software engineers, especially later in the platform's commercial lifetime.

As the Nintendo 64 reached the end of its lifecycle, hardware development chief Genyo Takeda referred to the programming challenges using the word hansei (Japanese: 反省 "reflective regret"). Looking back, Takeda said "When we made Nintendo 64, we thought it was logical that if you want to make advanced games, it becomes technically more difficult. We were wrong. We now understand it's the cruising speed that matters, not the momentary flash of peak power."

Texture cache

A challenge is presented by the texture cache small size of only 4 KB. This leads to developers needing to stretch small textures over a comparatively larger space. The console's bilinear filtering only blurs them. Due to the design of the renderer, when mipmapping is used, the texture cache is effectively halved to 2 KB. Toward the end of the Nintendo 64's market cycle, certain developers innovated with new techniques of precomputing their textures, such as the use of multi-layered texturing and heavily clamped, small texture pieces, to simulate larger textures; and with the streaming of precomputed textures into the small texture cache from the large, high speed, cartridge medium. Examples of this ingenuity are found in Rare's Perfect Dark, Banjo-Tooie, and Conker's Bad Fur Day and in Factor 5's Indiana Jones and the Infernal Machine. Some games use plain colored Gouraud shading instead of texturing on certain surfaces, especially in games with themes not targeting realism (e.g., Super Mario 64).

The big strength was the N64 cartridge. We use the cartridge almost like normal RAM and are streaming all level data, textures, animations, music, sound and even program code while the game is running. With the final size of the levels and the amount of textures, the RAM of the N64 never would have been even remotely enough to fit any individual level. So the cartridge technology really saved the day.

— Factor 5, Bringing Indy to N64, IGN

Fill rate

It may be that most Nintendo 64 games are actually fill-rate limited, not geometry limited; thus, there is a variety of possible techniques by which to maximize the fill rate. The RDP's (Reality Drawing Processor) fill rate is significantly affected by the optimizations taken in the microcode used by the developer — often specifically with Z-buffering. Thus, for maximum performance, the microcode supplied by Nintendo may be replaced by each developer. There has been controversy over the Nintendo 64's polygon per second rating of about 100,000; however, some of the most polygon-intense Nintendo 64 games, such as World Driver Championship, Turok 2: Seeds of Evil, or Indiana Jones and the Infernal Machine frequently push past the Sony PlayStation's typical in-game polygon counts.

Microcode

The graphics and audio co-processor is programmable through microcode. By altering the microcode run on the device, a developer can command different operations, create new effects, and optimize for speed or quality; however, Nintendo was unwilling to share the microcode tools with developers until the end of the Nintendo 64's life-cycle. Programming RSP microcode is said to be quite difficult because Nintendo's official code tools are very basic, with no debugger and poor documentation. As a result, it is very easy to make mistakes that are hard to track down, which could cause seemingly random bugs or glitches.

SGI's default microcode for Nintendo 64 is called "Fast3D", which some developers noted was poorly profiled for use in games. Although it allows more than 100,000 high accuracy polygons per second, this microcode is optimized more for accuracy than for speed, and performance may suffer as a result. Nintendo's own "Turbo3D" microcode allows 500,000–600,000 normal accuracy polygons per second. However, due to the graphical degradation, Nintendo officially discouraged its use. Several companies, such as Factor 5, Boss Game Studios and Rare, were able to write custom microcode that reportedly runs their game engines better than SGI's standard microcode would.

One of the best examples of custom microcode on the Nintendo 64 is Factor 5's N64 port of the Indiana Jones and the Infernal Machine PC game. The Factor 5 team aimed for the high resolution mode of 640×480 because of the crispness it added to the visuals. The machine was said to be taxed to the limit while running at 640×480, so they needed performance beyond that which was provided by Nintendo's standard SGI-designed microcode. The Z-buffer could not be used because it alone consumed the already constrained texture fill rate. To work around the 4 KB texture cache, the programmers came up with custom texture formats and tools to let the artists use the best possible textures. Each texture was analyzed and fitted to best texture format for performance and quality. They took advantage of the cartridge as a texture streaming source to squeeze as much detail as possible into each environment and work around RAM limitations. They wrote microcode for real-time lighting, because the supplied microcode from SGI is not optimized for this task, and because they wanted to have even more lighting than the PC version had used. Factor 5's microcode allows almost unlimited realtime lighting and significantly boosts the polygon count. In the end, the Nintendo 64 version of the game is said to be more feature-filled than the PC version, and is considered to be one of the most advanced games for Nintendo 64.

Factor 5 again used custom microcode with games such as Star Wars: Rogue Squadron and Star Wars: Episode I: Battle for Naboo. In Star Wars: Rogue Squadron, the team tweaked the microcode for a landscape engine to create the alien worlds. For Star Wars: Battle for Naboo, they used what they learned from Rogue Squadron and made the game run at 640×480, also implementing enhancements for particles and the landscape engine. Battle for Naboo has a long draw distance and large amounts of snow and rain, even in high resolution mode.

Memory

Featuring a non-unified memory architecture, the console's RDRAM has very high access latency, which is contrasted with its high bandwidth advantage. The R4300 CPU's ability to access RAM is constrained by its requirement to route all RAM accesses through the RCP, and by the fact that it can not use DMA to do so.