Privateer 2 editing and extracting

Discussion in 'General Wing Commander Chat' started by Lin Kuei Ominae, Feb 16, 2011.

  1. Bandit LOAF

    Bandit LOAF Long Live the Confederation!

    Messages:
    28,364
    Likes Received:
    524
    Trophy Points:
    69,485
    Gender:
    Male
    Location:
    Ashton, MD, USA
    Incredible! Thank you so much. It will certainly be a big help, since the version at WCNews was typed up by hand from screenshots by me and is full of typos. :)

    Please let me know if you come across...

    - The mission text (e-mails/briefings/etc.)
    - The variables that can slot into the booth entries. I really want to know what all the possible names for reporters are. :)

    Amazing, amazing stuff... but I'm sure I've told you that before.
     
  2. Lin Kuei Ominae

    Lin Kuei Ominae Spaceman

    Messages:
    65
    Likes Received:
    0
    Trophy Points:
    0
    With the p2-editor you can find them out too. (or look in the boothinfo.ini of the editor, where i collected already many different strings/names)

    entry 30 and up seem to be not used anymore as reporternames ingame as these represent strings for other things.
     
  3. Bandit LOAF

    Bandit LOAF Long Live the Confederation!

    Messages:
    28,364
    Likes Received:
    524
    Trophy Points:
    69,485
    Gender:
    Male
    Location:
    Ashton, MD, USA
    AWESOME, thanks!

    It's mostly Privateer 2 developers!
     
  4. HCl

    HCl WC EXE archaeologist

    Messages:
    327
    Likes Received:
    52
    Trophy Points:
    22,935
    There you go! ;) Those were actually scattered across several files. I'm not completely sure why at this point, but i'd guess it could be due to the game's mission selection mechanism (in order to give you a set of missions at various stages of the plot progression).

    Anyway, that should be all mission-related text. Hope it is useful!
     

    Attached Files:

  5. HCl

    HCl WC EXE archaeologist

    Messages:
    327
    Likes Received:
    52
    Trophy Points:
    22,935
    Managed to make a bit more progress on text extraction. Here's a few more bits (attached):

    - Ran into the Wingmen descriptions chunk. It was huffman compressed, but I don't think I posted it before, so here it is, in case it is useful.

    - Also, I finally managed to crack that dictionary-based compression algorithm I mentioned a while back! This gives us access to the Equipment descriptions, as well as a lot of compressed SHP chunks. I'm guessing this was adopted as a lighter alternative to deflate at some point during development.

    A couple references from this chunk might correspond to some ships that appear on various missions (eg. Police Ship Sweeney, S.S. Amelia, ...)

    - Finally, a couple of deflate-compressed chunks with miscellaneous data. These are not very interesting by themselves, but could be of some use later on (cross referencing some entries with other text files may be useful at some point, figuring indexes, and so on).

    Anyway fun stuff! :)
     

    Attached Files:

  6. Bandit LOAF

    Bandit LOAF Long Live the Confederation!

    Messages:
    28,364
    Likes Received:
    524
    Trophy Points:
    69,485
    Gender:
    Male
    Location:
    Ashton, MD, USA
    I think you may have discovered something extra-special here!

    What I'm betting it is is some earlier index for the database. Check out the list (below)--you've got planets, businesses, characters and what seem to be names of ships... but they don't match up to those lists in the game itself. Many that are in the game aren't listed while others we've never heard of are (Mehran Anderson? Simon Blade?)... and the ships all have names instead of registry numbers.

    The ships don't match missions... it seems to be a collection of fighter types from the game and entirely new names... plus a few of the flavor text names that wound up as classes in the finished database (Salvia, Yackard).

     
  7. HCl

    HCl WC EXE archaeologist

    Messages:
    327
    Likes Received:
    52
    Trophy Points:
    22,935
    Good catch, you may actually be right! I initially suspected there could be a few markers at the Mission text used for inserting a random ship name, so i dismissed the names I did not recognize due to simply not remembering them (not having played the game for some time).

    Turns out that ship names on the missions are actually hardcoded and, as you said, a few of the names do not seem to appear anywhere else, which means that the strings on this list may be from a previous version indeed. Exciting stuff! :)

    I wonder if other such artifacts are in the game files somewhere, waiting to be found... I'll keep looking!
     
  8. HCl

    HCl WC EXE archaeologist

    Messages:
    327
    Likes Received:
    52
    Trophy Points:
    22,935
    It turns out that the actual ship names and IDs in each mission are deflated in a DATA chunk in the MISSIONx.IFF files. I'm attaching these, in case it is useful. Indeed, you will find a few familiar names there:

    Can't say i recall these Papogod references, but i'm sure they appear in some of the Papogod-related missions. Anyway, hopefully these will be useful references for the WCPedia!

    Haven't run into any more text in the meantime. I made a script to scan the files for more chunks compressed with the newly-discovered dictionary-based compression algorithm, but there only seems to be one text chunk compressed with that algorithm. Everything else seems to be sprite data, 1.40 mostly.

    I think that should be most, if not all, of the text. Let me know if I missed something though!
     

    Attached Files:

  9. Dundradal

    Dundradal Frog Blast the Vent Core!

    Messages:
    6,469
    Likes Received:
    145
    Trophy Points:
    51,985
    Location:
    Seattle, Washington, USA
    HCl you continue to absolutely astound me good buddy!

    All this stuff I know is going to be put to good use in Lin's editor and in WCPedia!
     
  10. arfgh

    arfgh Spaceman

    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    50
    Gender:
    Male
    Hey guys.

    I am a Privateer 2: The Darkening enthusiast since its releasing in 1996.

    I played my version tons of times, but i can say that always was a big problem for me, because the game crash and crash and always crash when certain situation happends. This situation is when some enemy speak when we are shooting him, just when he die.

    With the time i have located the problem that is 100% sure into the file 'SPEECH.BIG'. My file contains audio spanish cause my version is this. If i remove this file from the installation, or just change it for the english or german version, the game never crash.

    In my opinion, the problem happends because the game dont stop to play the sound at its end, because sometimes i can hear a piece of a next audio communication, or the computer voice just repeat itself. The crash never happend in this moment, but the end of the mission will always finish with a crash. This happend with all the versions of the game, including the Windows deluxe version. In the past, when played the dos version, the whole pc crash, but now with the windows deluxe version, the game just stop and we are rejected to desktop with the dark.exe process in memory.

    I want to request to all, and specially to Mario Brito, the examination of my 'SPEECH.BIG' to know what is the problem. My objective is to reconstruct this file, with audio files that dont cause the problem. I know that Mario made a tool to extract the files into the .big, but i dont know how to import again files to a new 'SPEECH.BIG'.

    Download the famous 'SPEECH.BIG' --> here! it comes directly from my original CDs


    Plesase help me to fix this problem for one of my favourite games ever.
     
  11. thp

    thp Rear Admiral

    Messages:
    20
    Likes Received:
    3
    Trophy Points:
    21,885
    Location:
    Austria
    Really useful thread, thanks :) I think it was never mentioned what the algorithm for decompressing the non-Huffman, non-Deflate compressed data is. This is the data that starts with the bytes 0x10, 0xfb, and it looks like this algorithm is documented here (have yet to test it myself on the Privateer 2 data yet, but the data looks very much like it fits that description): http://simswiki.info/wiki.php?title=Sims_3:DBPF/Compression
     
  12. thp

    thp Rear Admiral

    Messages:
    20
    Likes Received:
    3
    Trophy Points:
    21,885
    Location:
    Austria
    Yep, this decompression algorithm works indeed. Not sure if it was mentioned already in this thread, but for image data ("1.40" signature), the shp2bmp code works, but if the first sub-image size is 3 * 256 (768 bytes), then this has to be interpreted as a 24-bit RGB palette for the other images (there might be some info in the header to determine whether it's a palette, but the 768 heuristic worked fine so far). Not all images contain a palette, for those that don't (e.g. pilot comm images), indexing from GAME.GIF should work.
     
  13. thp

    thp Rear Admiral

    Messages:
    20
    Likes Received:
    3
    Trophy Points:
    21,885
    Location:
    Austria
    Some progress on extracting models; the SPACEGEO.IFF file refers to files in SPACETEX.IFF (obviously). The VERS chunk contains 10 floats per vertex, the first 3 are the (x, y, z) coordinates, the next 2 are the (u, v) coordinate. The last three are likely normals (the vector length is 1.0), but haven't gotten them to work correctly, so the normals are not yet used. The FACS chunk contains 18 uint16_t values per face, the first 3 are indices into the vertex table. Also the "_d1", "_d2", "_d3" suffixes specify the LOD levels for the models (with _d1 being the most detailed, and _d3 being the least detailed). Here's some random models imported into Blender.
     

    Attached Files:

    delMar likes this.
  14. AD

    AD Finder of things, Doer of stuff

    Messages:
    4,931
    Likes Received:
    104
    Trophy Points:
    36,885
    Location:
    BC, Canada
    Hi thp! It's pretty cool to see you taking a crack at this and resurrecting this thread.
     
  15. thp

    thp Rear Admiral

    Messages:
    20
    Likes Received:
    3
    Trophy Points:
    21,885
    Location:
    Austria
    Cleaned up the 3D model extraction code a bit, and fixed a UV coordinate calculation bug that caused artifacts with the planet textures.

    Here's a quick'n'dirty WebGL-based Model Viewer for the extracted models, which were converted to Wavefront OBJ, and colormaps to PNG:
    http://originalthp.solsector.net/p2models/
     
    AD likes this.
  16. AD

    AD Finder of things, Doer of stuff

    Messages:
    4,931
    Likes Received:
    104
    Trophy Points:
    36,885
    Location:
    BC, Canada

    Awesome work! HCl and I (mostly HCl really) did some work making a viewer for the CIC website for the WC3/4/armada ships that uses webGL. We had talked about expanding it to include the formats used in some of the other games at some point. I have no idea if it's actually linked on the CIC front page but the viewer can be found here http://www.wcnews.com/shipviewer/ and the update here http://www.wcnews.com/news/2014/08/17/hcl-s-online-ship-viewer

    I know we had been looking into the P2 format a bit at one point but I don't really remember how far HCl got with extracting the actual ship models. However, it might be useful for comparison purposes, but Paul Hughes sent in a batch of source 3DS models for P2 at one point http://www.wcnews.com/news/2009/08/11/privateer-2-in-3d

    EDIT: Quick comment on your model viewer... great work. Pretty much every model has three LOD versions, which is neat. Also, I know that it's a "quick and dirty" version, it does look like some of the palettes aren't being displayed correctly on some objects and ships. Possibly other transparency information isn't working correctly either... I could be wrong, but thinks like the sheild hit tiles look funny. With the WC3 and 4 stuff, there were also issues with objects that had pretty much zero thickness and showing correctly since sometimes they needed to be told to display on both sides since they would be invisible if you were looking at the back of the object.
     
    Last edited: Oct 15, 2016
  17. thp

    thp Rear Admiral

    Messages:
    20
    Likes Received:
    3
    Trophy Points:
    21,885
    Location:
    Austria
    You can extract the files yourself with the tool below (point it to SPACETEX.IFF and SPACEGEO.IFF). I tried the .3DS files in Blender, but didn't get textured models (but that might be a limitation of the Blender 3DS importer).

    Here's a link to the tool I've been working on in the last few days based on this thread, it can extract quite a bit of Privateer 2 data, but there are still some known bugs (e.g. it doesn't apply the palette contained in the BASE image to the animation images in SETS.IFF, so the palette is off there -- but the animation images work great with the BASE palette; the same is true for some images in BOOTH.IFF -- the palette is available inside the file, just not applied everywhere):

    http://originalthp.solsector.net/priv2dump/

    I have tested it with the German and English version of Privateer 2; if anybody could test it with other language versions, that'd be great.

    Random observation: If you dump BOOTH.IFF, you'll notice that the pre-rendered 3D rotation images of ships and equipment only contain half a rotation -- I first wondered why that is, but then noticed that all ships and equipments are symmetrical, so a full rotation can be done by doing the half rotation, and then doing the other half of the rotation by mirroring the image horizontally and playing it back in reverse.

    There are texture maps that convert certain colors in the palette to some other colors, these probably need to be applied properly depending on ship type (maybe some ship models are also re-used and just applied with different colors). Right now, all the colormaps from SPACETEX.IFF are written using the default (GAME.GIF-based) palette - this might explain some wrong textures you see. Transparency is easy to fix, right now the tool doesn't apply any transparency, but it's possible to make palette index zero transparent (which seems to be the way the game handles transparency also in other images), which fixes the transparency (but some images such as BASE images don't have transparency, so I didn't want to apply that unconditionally).

    The "zero thickness" issue (e.g. seen with the missiles) is just backface culling being turned on in Three.js by default, if one would turn that off, it would look correct (possibly somewhere in the face definition it would tell us if it's a two-sided triangle, or we just turn it off unconditionally).
     
    AD likes this.
  18. AD

    AD Finder of things, Doer of stuff

    Messages:
    4,931
    Likes Received:
    104
    Trophy Points:
    36,885
    Location:
    BC, Canada
    This is an awesome tool, thanks! I had no trouble with the unpacker at all. I haven't spent a ton of time playing with it yet but I did find one fun thing... looks like the menu screen for some kind of debug/programming tool

    TESTBED.SHP-1.png
     
  19. thp

    thp Rear Admiral

    Messages:
    20
    Likes Received:
    3
    Trophy Points:
    21,885
    Location:
    Austria
    Interesting, I didn't find this in my version of Privateer 2 -- which version and language are you using / in which file did you find that image?
     
  20. AD

    AD Finder of things, Doer of stuff

    Messages:
    4,931
    Likes Received:
    104
    Trophy Points:
    36,885
    Location:
    BC, Canada
    I have the English DOS CD version. I'm fairly certain it's the US version. I believe the file was TESTBED.SHP and was on CD1.
     

Share This Page