Pedeca

For the engineers out there who has been wanting to figure out how to modernize the Westwood 3D and/or Strategy Action Game Engine, this is for you. I have decided to impart all that I've scrounged up on both engines in the hopes it will be useful to someone in the community at large. A brief history of WW3D Engines begin somewhere, and for Westwood Studios' first 3D engine, they built theirs off an existing 3D engine. That engine was the SurRender 3D engine by Hybrid Graphics (now Umbra under Amazon). That was a 5 year odyssey that started in 1997 with Project "Commando" and ended (after two delays) in 2002 with C&C Renegade. The longest game development in Westwood Studios history (C&C Tib Dawn and Tiberian Sun were each 4 years by comparison), but, to be fair, it was both Westwood's first fully 3D project and first person shooter (and a hybrid one at that). When Westwood Pacific (EA Pacific/EALA) was looking to make a new 3D engine for RTS games, they considered multiple engines such as id Software's id Tech 3 (the "Quake III engine"), Epic Games' Unreal Tech Engine (likely Unreal 2), and Monolith Productions' LithTech, but settled on an in-house engine they already owned for SAGE. After Westwood was shuttered and reborn as Petroglyph Games, they too built their Alamo Engine based on the Westwood 3D engine, and then their modern GlyphX Engine after that. Yes, GlyphX is the most modern implementation of the WW3D engine, though GlyphX is more like a fusion of the capabilities of WW3D and SAGE/Alamo. So it's for more than RTS games. So, with that out of the way, let's take a look under the hood. The Westwood 3D Engine Genealogy At A Glance It's best to start with a visual aide to see how these engines branched off and what Application Programming Interface (API) they use. Attached is a simple flowchart I made that tracks the evolution of the engine from SurRender 3D all the way to SAGE2 and GlyphX. If the image (for whatever reason) never loads or is lost in the future, the genealogy looks like the below: [SurRender 3D (OpenGL)] -> [WW3D (DirectX8.1)] -> [SAGE 1 (DirectX9)] & [Alamo Engine (DirectX9)] -> [SAGE 2 (DirectX10)] & [GlyphX Engine (DirectX10, 11, 12)] Yes, that is correct, the engine originated with the OpenGL API before immediately switching to the DirectX API. This was done because Westwood had more experience working with the Windows API than others at that point. This could suggest that the main game code for WW3D and SAGE is API agnostic, but there is much of the code that was built for MS Windows programs which entails extra work to create something that also works on other OS's like Unix/Linux (though Steam already wraps all games in a wrapper that allows them to run in the Unix/Linux environment regardless). The Graphical API (the Construction Yard of the engine) So with that, the first and most obvious improvement for the engine lies with the graphical API. Renegade was built on DX8.1, the same API that the DirectX Box (Xbox OG) was built on and C&C Generals was built on the next iteration with DX9, the same version of the API used for the Xbox 360. Now, most of you are probably chomping at the bit, wanting to use the most recent iteration of DirectX as the chosen upgrade. I do not advise that. DirectX12, like with Vulkan, is a low level graphical API that is built with C, unlike older DirectX versions and OpenGL, which both are built in C++. So there is a whole learning curve to overcome that even some developers have struggled with. Hence, the best choice would be DirectX11 since it's still in C++ and you can use it as a fallback renderer for when you do choose to implement DX12 (like most companies do). A similar dual API technique is done with OpenGL and Vulkan (like how id Software implemented id Tech 6). With the API upgraded, we can have access to many modern features, such as superior texture compression (BC6 and BC7) and material shaders in line with the most recent version of WW3D (GlpyhX). Although DX8.1 and DX9 do have texture compression (via DXT) and can do material shaders, they are comparatively primitive and restrictive. So if you choose to continue to use the old API, I'm just letting you know that you can still do some advanced tricks. All versions of DirectX can be downloaded directly from Microsoft's site. OpenGL and Vulkan can be downloaded form Khronos Group's site. All can be integrated within Visual Studio. All necessary for compiling new .dll files for the engine. Some DX11 samples and tutorials: DirectX SDK Samples Wiki RAD Tools and Their RADical Uses (and their replacements) RAD Tools were some of the most popular proprietary engine tools of their time.... that was until Epic Games bought them out and they were surpassed in multiple aspects. Thus it's not all too necessary to replace all of them, but one in particular will need an urgent replacement. WW3D and SAGE both use them, so let's evaluate each one individually. BINK Video (for Cutscenes and EVA Unit videos that play in the Radar) When you need to compress a video file so that it fits on a CD/DVD disk, you either have to create your own method of compression, or use someone else's. For many, that solution is BINK Video. You've probably seen the little twister logo on many a game splash screen on start up, well I'm here to tell you that BINK is for real time video compression and file reading. Like a file zipper/unzipper for video files. BINK comes in two versions: BINK 1 (.bik files) and BINK 2 (.bk2/.bik2 files). BINK 1 is what all WW3D, SAGE 1, and Alamo games used for their video files, while SAGE 2 and GlyphX started using BINK 2. The major difference between the two versions of BINK is that the second edition requires that your engine's Math library contains SSE (specifically SSE2 or later if I recall correctly) in order for it to work (oh and BINK 2 is of a higher quality and compression than BINK 1). That's not a calculation that either WW3D nor SAGE can do (as they have the same WestWood Math library). This will be expanded in a later section regarding the Math Library and Physics Library. So if you want to continue to use BINK, you're in luck! It's still the industry standard for video compression (found natively in Unreal and Unity) so it's not too difficult to implement. There is, however, one catch: BINK is propriety software, and thus you will have to pay for it if you choose to sell your game/project on the market. If not? Well then you can download the tool and SDK and use them as you please, but do be mindful that if you are to go commercial, you will need a license to continue using it. The running price for a BINK license is $11,000 USD per game and per platform. No free lunch. [BINK Link] If that's too many credits, then I do have an alternative for you open source adherents and it's the WebM format! It's open source, free to use, and offers superior compression than BINK 2! However, you will have to adapt the source code to replace the all the BINK code references with WebM (and update the Math Library all the same). Not too hard, but work all the same. Companies like Valve have replaced BINK with WebM in Source 2, so there's your big dev implementation. [WebM site (go to Codec for source code)] The Miles Sound System (for epic SFX, Spacial Audio, and the Soundtrack output) [DEFUNCT] Westwood Studios games, and EALA by extension, are well known for their awesome audio. Great sound mixing, rich SFX, and even 3D audio for that EAX (Environmental Audio Extensions) experience that allows for echos and accurate audio depiction. This was all accomplished in WW3D and SAGE with the the Miles Sound System... which has been retired as of 2022 and is no longer offered nor downloadable from RAD Tools. This also means updating the Assimp3 code (misspelled as "Asimp3"). [Assimp] That's short for "Asset Importer," stop laughing. However, modern audio implementation may have better solutions to use for asset importing. Thus a replacement is necessary. So here are some great alternatives: MSS Alternatives List System Price External Links Wwise Depends on the license ($0-$40K) Wwise (Audiokinetic) FMOD FREE FMOD Steam Audio FREE Steam Audio Source Code (Valve) OpenAL FREE OpenAL OK, this needs a bit of explaining. Wwise and FMOD are the audio tools that you use to make your soundscapes, SFX, and everything audio that will go into your game (formatting and such). The MSS replacement. Steam Audio and OpenAL are the source code that implements that audio in real time for your game. WW3D and SAGE use Westwood Audio for implementing audio, but it will need to be adjusted to work with Wwise or FMOD and those source code libraries have the code to implement that change (handy!) Why Wwise? It's the best audio tool on the market used in all the biggest games. It has a variable license depending on the size of your project (which also means variable access to the program's capabilities). You can download the Wwise program free like the programs and source codes on the list, you will just have to apply for a license when it comes to making anything you intend to make money with. Otherwise (heh) it's free to use for small scale stuff. FMOD is still there if you need it. Oodle and Telemetry (the other tools you may or may not need) Oodle is a file size optimizer tool that helps you keep your game as small as possible. If you have ever seen the octopus Oodle logo on a game splash screen, that's what it's there for. There are alternatives like ZLIB (which SAGE uses, not a bad thing to add to WW3D), so you have options. Oodle costs $11,000 USD per game per platform. [Oodle Link] Telemetry is a tool that helps you optimize your game by showing you how it's performing in real time (you know, like showing you the Frames Per Second count). SAGE natively used BYTEmark for telemetry (we'll talk about this again later). There are others out there, including natively in-built telemetry tools in engines such as id Tech and Panda3D if you want to borrow other's tools via source code. And of course there are likely many others on the market to choose from. Telemetry costs $9,500 USD per game per platform. [Telemetry Link] As a reminder, both Oodle and Telemetry, as with any RAD Game Tool, are free to use so long as you are not charging anything for your game. NVidia functions (our first obsolete code!) [OBSOLETE] NVidia, when they got into the video game ecosystem, added new features that helped support the industry. Those tools and source code were distributed by NVidia for use by every company. They were so useful that they were integrated directly into Graphical APIs. Which means that the code that WW3D includes from NVidia is completely obsolete as it's handled natively by the DirectX API (and even OpenGL and Vulkan). It really shows WW3D's age and how long it took to develop Renegade. NvDXTLib (the oldest way to implement real time texture compression) [OBSOLETE for WW3D] Back before DXT compression was added natively to DirectX, this library in combination with NVidia's online tools were the only way to implement real-time texture compression. Since at least DirectX9, this is no longer needed. Just remove it from the WW3D code like SAGE did. Any modern API like DirectX11/12 and OpenGL/Vulkan have native texture compression built in, so you will never have to bother with this part of the source once you upgrade the graphical API. NVASM (HLSL the old fashioned way) [OBSOLETE for SAGE] Once again, back before the High-Level Shader Language became native, this was the only way to add the component to APIs like DirectX8.1. "But wait! SAGE is built on DirectX9!" I know, it's really weird to see this in SAGE given that API base. Perhaps it wasn't native to DX9 at the time, but it was definitely soon made obsolete by later versions of the code. So, yet again this is the case of a useful code base provided by NVidia that became a standard feature in modern graphical APIs like DirectX, OpenGL, and Vulkan. You can remove this code as soon as you update SAGE's graphical API. Lightscape and 3DSMax (3D asset importers, and what to do about them) All those who work with WW3D and SAGE know about the Westwood3D (.w3d) model format, but video game engines also include other asset formats to maximize the use of the modelling programs that they use to create 3D models in. For WW3D and SAGE, the other kind of assets include the Maya Binary (.mb) and 3DSMax (.MAX) model formats for static objects (though I cannot be certain if WW3D also had GMAX file compatibility, not that you would want it). Lightscape was an old software that Westwood used for making environments in Renegade with the lighting included. This was to import assets from programs like GMAX (the old game specific version of 3DS Max) without needing any conversion. SAGE replaced Lightscape with 3DSMax's asset importer. Later, SAGE 2 seems to have replaced 3DSMax with Maya (as shown with the behind the scenes for Tiberum Twilight). When it comes to source code involving 3DSMax and Maya, you will have to need to contact Autodesk directly to get that and possibly a license. Not much you can do about that. Unless, of course, you decide to use other model formats. You have a great variety of model formats to choose from: Filmbox Object (.fbx), Wavefront Object (.obj), LightWave Object (.lwo, Westwood made their 3D animated cutscenes in LightWave3D), COLLADA (.dae "digital asset exchange"), Extendable 3D (.X3D), Universal Scene Descriptor (.usd or .usdz), and gITF (.gLTF and .glb). So you have options outside of Autodesk's proprietary formats to use in your version of WW3D or SAGE (and you can just use as many as you want! The sky's the limit!) Many of these formats have their importer/exporter source code online. Do note that only some formats include animation data like .W3D, such as .lwo (hence why id Software's id Tech engine still use it alongside .md6), .X3D, .usd/.usdz, and .gLTF/.glb. Knowing that, it begins to make sense why companies made their own 3D formats to contain both 3D model data, animation data, and other properties like for particle emission. BYTEmark (aka NBench, the BYTE Magazine benchmark tool) OK, now we can talk about SAGE's telemetry. SAGE added BYTEmark (aka NBench) to the code for checking the performance of the game while it's operating. It's still very usable for WW3D and SAGE when it comes to testing your levels. You can get it here: [BYTEmark Link] [NBench Link and others]You can read more here: NBench (aka BYTEmark) BYTEmark has the following features: Numeric sort - Sorts an array of long integers. String sort - Sorts an array of strings of arbitrary length. Bitfield - Executes a variety of bit manipulation functions. Emulated floating-point - A small software floating-point package. Fourier coefficients - A numerical analysis routine for calculating series approximations of waveforms. Assignment algorithm - A well-known task allocation algorithm. Huffman compression - A well-known text and graphics compression algorithm. IDEA encryption - A relatively new block cipher algorithm. Neural Net - A small but functional back-propagation network simulator. LU Decomposition - A robust algorithm for solving linear equations. However, BYTEmark is outdated as a telemetry and has the following shortcomings: It cannot access the GPU (it was made when the CPU was still the common processor for graphics in the 1990s). These benchmarks are meant to expose the theoretical upper limit of the CPU, FPU, and memory architecture of a system It cannot measure video, disk (not like we're distributing on disks, more on that later), or network throughput (those are the domains of a different set of benchmarks) NBench (BYTEmark) is single-threaded. Currently, each benchmark test uses only a single execution thread. However, most modern operating systems have some multitasking component. How a system "scales" as more tasks are run simultaneously is an effect that NBench does not explore. So rather out of date with modern architecture So, if you seriously want to make a modern game on WW3D or SAGE, you will need a modern telemetry tool that can access modern hardware and networking. There are multiple available telemetry options available. The aforementioned Telemetry by RAD Tools, but there are also others out there that are specific to certain metrics (like networking) that can be used in tandem with BYTEmark. Maybe the community will create some kind of new solution out of existing telemetry code bases. This one is definitely a undertaking knowing how many WW3D and SAGE mods are multiplayer-centric. Might not hurt to ask what either Petroglyph or EA uses for their games. STLport (the multiplatform ANSI C++ Standard Library) [OUTDATED] STLport was a C++ library extension first introduced open source back in 1997. It's a library of standardized operations that work across multiple platforms, similar to Microsoft's STL library, but faster. There's a lot of useful tools in the library, such as those for networking. This was added to SAGE and likely was still used in SAGE 2. The version SAGE 1 used was STLport 4.5.3, but you'll want the most latest release of STLport which you can get here: [STLport 5.2.1] You can also get the older STLport that SAGE 1 used here [STLport 4.5.3] However, STLport is antiquated. Users started using alternatives since before 2014, and its last update was in 2019. Some users reported that it's not compatible with C++17 standards or above. That's not good when you're modernizing the source code. So this too will need to be replaced with a more modern implementation. Lucky for us there are many available to choose from. First up is Electronic Arts' very own EASTL library. The very same used in the Frostbite Engine: [EASTL source] No doubt if there ever was a SAGE 3 it would have used it, just like the Generals 2 project was using it (because it was using Frostbite). This library is up to date and maintained by EA, so that works for us. Another replacement option is the Boost library [Boost Library] which could be a possible consideration. Most users of STLport replaced it with CXX [CXX Tools] which might work as well. Lastly there is also the Qt library [Qt Base] which might also be considered. Umbra Occlusion Culling (how the camera renders what's in view) WW3D inherits SurRender3D's Umbra occlusion culling solution for the game camera's frustum in-view rendering. That means WW3D uses Umbra to render what's visible and not render what's not visible [Umbra Occlusion Culling Explanation]. Umbra is still used in the industry, but it's proprietary under Umbra (formerly Hybrid Graphics) and owned by Amazon. This means you would have to get in contact with Amazon if you want to use Umbra via a license. Umbra is still used as a middleware for modern implementation of occlusion culling (even id Software used it) so it's worth checking out if you can get a license. [DOOM2016 Graphics Study] SAGE is the evolution of WW3D, but it does not use Umbra, likely to avoid royalties, but also because new solutions were quickly coming about that were more useful for RTS games. That said, SAGE's occlusion culling is the obvious replacement for Umbra, but we shouldn't stop there. Newer occlusion culling solutions have appeared that supersede older methods due to superior rendering speed and efficiency. Doom 2016 uses the highly efficient Clustered Forward Rendering method that might be a perfect natural upgrade for WW3D and SAGE (using SAGE's code as the base). In the meantime, it might be just as useful to borrow id Tech's occlusion culling code for id Tech 4 as well [id Git for all open source id Tech][modernized Doom 3 BFG using Vulkan and DirectX12] Bonus: here's a solution someone made to combat wall hacking [Corner Culling anti-hack]. Might be useful for those who work with competitive multiplayer. GameSpy SDK (RIP GameSpy) [DEFUNCT] When GameSpy went down, it took all the multiplayer servers with it. C&C was heavily dependent on it for providing reliable multiplayer. As such it will have to be replaced in order for multiplayer to function. Luckily, GameSpy was used with so many games that there were multiple replacements made for it. You have options to consider: OpenSpy [OpenSpy client code] UniSpy [UniSpy SDK] Valve's Steam Networking SDK [Game Networking Sockets SDK] Finally, there's also working with the C&C Online team to fully integrate their C&C network hooking solution into the source code directly. [CNC Online] GNU Regex (open source regular expressions) [OUTDATED for WW3D] WW3D uses the GNU Regular Expressions library for its code. This particular library was likely used for the POSIX functions, which are now fully integrated with modern C libraries. Yet another thing that shows the age of WW3D. You can remove this code just like EALA did for SAGE's implementation. What's POSIX? Well it's more for UNIX/Linux compatibility [What is Posix?] SafeDisk API (Remember when you had to enter the CD Key?) [OBSOLETE] Both WW3D and SAGE used to use SafeDisk to prevent illegal copying. By Windows 7, this API was on its way out needing backwards compatibility and by Windows 10 it is no longer compatible. Honestly, with the rise of online game clients like Steam and GOG (and C&C fan sites), are we really selling computer games on disks anymore? You can completely remove all the code related to the SafeDisk API in WW3D and SAGE without ever looking back. It's no longer needed. That aside, the API was proprietary and now it's no longer offered. Microsoft Cab Archive (for compressing data) [OUTDATED for WW3D] WW3D used the MS Cabinet Archive Library to compress files into the .CAB format. It's a bit different from zip files in compression method, but in the end it's for the same reason. You can find some code online for .CAB extractors, but SAGE replaced this with the ZLIB library to use zip files instead. So that's probably the best option as well. ZLIB (Zip files for gaming) [The SAGE Solution] SAGE uses the ZLIB library [ZLIB 1.3.1] and it's code for using it should be copied into WW3D. It's a logical upgrade. That aside, if you wish for superior file compression, Oodle from the RAD Game Tools is also available at different levels of compression. Though zip files should serve you decent until you start making your game rather large. LZH-Light Compression (additional compression option) [OUTDATED for SAGE] SAGE also includes the LZH-Light compression library [LZH-Light Code]. It's an old one and needs to be replaced because there are no recent releases for it and thus it's not compatible with modern C++ standards. [What is LZH-Light?] It's part of a family of LHA (originally "LHarc") archiving programs made by Haruyasu "Yoshi" Yoshizaki. Luckily LHA is still around, even if Oodle has replaced most compression methods for games, and you can get source code for it here: [LHAsa source] There are other versions out there as well for various OSes since the 90s, so there's an interesting assortment to dig around to find alternatives [LHA Wiki with further links] RTPatch Library (for updating the game via Patches) WW3D uses the RTPatch library for patching the game code. This is a proprietary library that was used by practically all the major companies back in the day. You might be able to get a modern implementation of it from Pocket Soft [Pocket Soft Contact]. Though it would appear that SAGE replaced it (as did many companies with their later engines), so it's a good bet that you could probably just use what SAGE uses. Or any modern patch solution really. The source code only exists to decode the RTP patch format. There is a modern patching solution that EA uses called Known Version Patching (KVP) [KVP] that makes use of Amazon services to do the patching [The Amazon Component]. I'm not sure if EA does licensing for it, but it wouldn't hurt to ask. Then there is the custom patching route. You can pay for a lifetime license to use a customizable game patcher for either WW3D or SAGE [Game Patch Creator] It's worth consideration. Java Runtime Headers (for online servers in WW3D) This is one is the easiest to do. Just grab the most recent Java Runtime Header library and add it in [Oracle Java Downloads] However, this is an old way of handling online. SAGE replaced this fully with its version of the GameSpy implementation, and with WW3D also having GameSpy probably means that the Java code was a transitional backup. All the same, it doesn't have to be removed, and can simply be there as backup still. Math and Physics Updates (Modernization Options for WW3D and SAGE) It's not necessary to update the Westwood Math and Westwood Physics libraries in order to compile your code, however, it is a major upgrade to get more out of the engine. You cannot use image compression like BINK2 without scalable vector graphics, which requires the SSE math function at version 2 or higher. [Image Processing Libraries that contain SSE4.2] <-(might be necessary if the below math libraries do not include SSE4.2) When EALA transitioned from SAGE 1 to SAGE 2 with Red Alert 3, they didn't just change the graphical API from DX9 to DX10, they also replaced the Westwood Math and Physics libraries with RenderWare's Math and Physics libraries. Unfortunately, Criterion's RenderWare Engine is still proprietary with currently no way to license it despite EA removing engine restrictions for their devs. So that means we need alternatives to either splice into the Westwood libraries, or to replace them. There are many options to choose from for C++ math libraries, so it's not that hard to find one that works. Here's a few: DirectX's XNA Math Library [XNA programming guide], OpenGL Math (Not just for OpenGL) [GL Math], Sony's Vector Math Library [Sony Vector Math], and Eigen (the math library that Panda3D uses) [Eigen Main Site]. As for physics, there are multiple solutions: Havok (outperformed by other libraries) [Havok Physics], Bullet [Bullet Physics 3.0], ODE [ODE Physics (old and used to fill gaps in Bullet)], and Jolt (doesn't have impulse, but has everything else) [Jolt Physics]. (Personally, I support using Jolt) The Final Hurdle for either WW3D or SAGE As described in the Renegade and Generals source codes (compiling section), you will have to update the .dsw file (old Visual Studio file) to the modern Visual Studio format. This will be a pain in the butt, however it is necessary. I wish the best to whoever has to do that process, because it will be slow. Anyway, thank you for reading this. I hope this ends up being useful to someone.