Privateer 2 editing and extracting

Discussion in 'General Wing Commander Chat' started by Lin Kuei Ominae, Feb 16, 2011.

  1. AD

    AD Finder of things, Doer of stuff

    Messages:
    4,918
    Likes Received:
    102
    Trophy Points:
    36,885
    Location:
    BC, Canada
    So I figured out what I was doing wrong with one of the programs I was using. I was specifying the offset incorrectly. Anyway, do a search for "offset file unzipper" and you'll get the command line program as well as the uncompiled source. This program lets you both unzip each def chunk into a *.dat file which is just the straight code of the chuck or give you the location of the zlib chunks. You can also unpack the zlib chunks into one file.

    When I run it on SPACE.IFF I get similar looking code to what HCl posted. Basically you use the '-a' option and specify the file you want, indicate your output directory... and then put in the offset for the first def chunk which is really easy to find using the VIFF tool. When you click on any individual chunk in the list in VIFF you will get a box showing that chunk's code. If the first three letters are DEF then record the offset that is in written in the top of the that particular window, else do a search for DEF in a hex editor.

    Here's a sample of what I got when I ran it on a texture file:
    [​IMG]
     

    Attached Files:

  2. HCl

    HCl WC EXE archaeologist

    Messages:
    327
    Likes Received:
    52
    Trophy Points:
    22,935
    AD, yep, that looks exactly the kind of hex-patterns i would expect to see on an 8-bit sprite/texture. Looks like that tool you found does the job very nicely as well!

    Alright, i uploaded the Visual Studio C++ project here.

    A couple of notes:

    - My VS project goes looking for zlib on the "c:\projects\zlib..." directory. This path is absolute, so you will want to go to the project's property pages and change it to your zlib path. Also, and this might be more or less obvious, but when compiling zlib be sure to compile without asm, otherwise you'll have to have Masm installed.

    - Since you're developing your tool in C#, i guess you'll need to interface with zlib using either .NET Interop or Managed C++ (the former is probably more straight-forward, but it really depends on how you're doing things on your end). Also, my project links zlib statically, but I am betting it will probably be easier for you to link it dynamically.

    Edit: actually, after hitting "submit", it occurred to me that there should already be a port of zlib for .NET out there. Sure enough, there is one, so you may want to take a look at it: http://www.componentace.com/zlib_.NET.htm
     
  3. AD

    AD Finder of things, Doer of stuff

    Messages:
    4,918
    Likes Received:
    102
    Trophy Points:
    36,885
    Location:
    BC, Canada
    So I took a look at MUSIC.IFF. I can say that the "offset file unzipper" works great but that I didn't really learn anything new. MUSIC.IFF has... a couple sections. THere's a BMSQ section with six BMSD subsections that would at a guess correspond to the six song files... possibly instrument or usage or soundfont info perhaps? That's followed by the six zlib compressed song files themselves in the BMTN section. Running the zlib unpacker extracts all six song files. If you used the program I mentioned then all you have to do is rename the file extension to XMI.

    The P2 ingame music is all XMI files which were some kind of proprietary extended MIDI format that can be a pain to find programs to play. Awave studio does it but it's one of those timed trial deals. There's probably a winamp pluggin out there or something though. Someone obviously extracted these one upon a time though as the files I ended up with are identical to the ones in the CICs files section (There's a zip with both the XMI's and versions converted to regular MIDI).

    I guess it would be possible to replace the inflight music with custom midi's but I would bet people would rather work out a way to have custom playlists of MP3s or something which is probably a lot harder to implement. On the otherhand, MIDI support in newer versions of windows is dwindling.
     
  4. Wedge009

    Wedge009 Rogue Leader

    Messages:
    10,056
    Likes Received:
    28
    Trophy Points:
    69,385
    Gender:
    Male
    Location:
    Sydney, Australia
    I find the MIDI plug-in included with the standard Winamp package is usually pretty good at playing even the most esoteric MIDI formats.
     
  5. AD

    AD Finder of things, Doer of stuff

    Messages:
    4,918
    Likes Received:
    102
    Trophy Points:
    36,885
    Location:
    BC, Canada
    Possibly. I found stuff suggesting winamp plays XMI files, yet couldn't find it in the latest version, though I refuse to pay for the pro version and it's possible they moved XMI support to the pro version. I really didn't look that hard for a plugin either.
     
  6. Lin Kuei Ominae

    Lin Kuei Ominae Spaceman

    Messages:
    65
    Likes Received:
    0
    Trophy Points:
    0
    Thank you HCl. As soon as i've find some time i'll give it a test and start creating a user friendly editor for the files.
    Also thanks for finding the .net zlib library.

    @AD: I just found this: xmi2mid

    a quote from a different forum (here)
     
  7. AD

    AD Finder of things, Doer of stuff

    Messages:
    4,918
    Likes Received:
    102
    Trophy Points:
    36,885
    Location:
    BC, Canada
    Cool! That worked, thanks. The only other thing I had to do was in the midi configuration switch volume control from auto to driver-specific. Otherwise it was doing some funky things to the sound that I can't quite explain.
     
  8. Wedge009

    Wedge009 Rogue Leader

    Messages:
    10,056
    Likes Received:
    28
    Trophy Points:
    69,385
    Gender:
    Male
    Location:
    Sydney, Australia
    If you found something that works for you, then good, but I assure you that you don't need to pay for Winamp to get the MIDI plug-in to play XMIs. I've used it to convert and/or play extracted MIDI data from fun DOS-era games like Syndicate and Lemmings Chronicles.
     
  9. Bob McDob

    Bob McDob Better Health Through Less Flavor

    Messages:
    3,390
    Likes Received:
    1
    Trophy Points:
    36,885
    Location:
    Grassy Knizzle
    Well, it plays, all right; but I can't say that the data that comes out sounds much like the music in-game.
     
  10. Wedge009

    Wedge009 Rogue Leader

    Messages:
    10,056
    Likes Received:
    28
    Trophy Points:
    69,385
    Gender:
    Male
    Location:
    Sydney, Australia
    It's dependent on what output format you're using, of course, as well as what MIDI-capabilities your hardware has available, if any.
     
  11. HCl

    HCl WC EXE archaeologist

    Messages:
    327
    Likes Received:
    52
    Trophy Points:
    22,935
    Speaking of which, i extracted some MIDIs back in the day as well, they're still in my site here.

    It's probably the same XMIs you just extracted, I remember i also used a similar xmi2mid tool to convert the files to a more supported format. The only thing noteworthy in my mind is that i extracted these before i figured out the Def! chunk compression, using a small asm utility for creatively dumping memory from the game at given points. For this reason, there might be a few midis that I missed.
     
  12. AD

    AD Finder of things, Doer of stuff

    Messages:
    4,918
    Likes Received:
    102
    Trophy Points:
    36,885
    Location:
    BC, Canada
    Did some experimenting with this. The samples in speech.big definitely sound best at 11025. I can't tell if the extra noise is on purpose but the files do sound noisy within the game. However, I found that the ones in DANI.BIG sounded best at 16000. DANI... not sure why it's called that but it seems to be the all the ship's computer audio.

    So, I was looking at some of the IFFs contained within the isets and sets IFF files. Curiously a large chunk of the ISETS files looked an awful lot like the FAT files so I ran them through a similar process... and wouldn't you know? The isets files are almost strictly the FMV choice dialogue. They played very clearly at 22050 khz. Looking at anhur_tf.IFF you can see four separate 'speech' chunks with associated 'text' chunks. This is somewhat corroborated by the audio itself which is basically four sets of choice dialogue (8 lines, 4 pairs)

    Now there's definitely also audio within the SETS.iff files as well but it's a much smaller portion of the file itself. Most of the file just plays like garbage but there's a distinct section at the start that I'm pretty sure are the background noises you hear when you are on a planet at... say, the customs screen. Some of the effects sounded a bit garbled at 22050 and others like the crowd chatter sounded very clear. Looking at something like anhur.iff I would guess the effects are in the very beginning FATF chunk. My guess would be that the rest of the file is hotspot info and animation frames for the various base 'objects'.
     
  13. Bandit LOAF

    Bandit LOAF Long Live the Confederation!

    Messages:
    28,355
    Likes Received:
    521
    Trophy Points:
    69,485
    Gender:
    Male
    Location:
    Ashton, MD, USA
    The Bitching Betty voice was voiced by actress Dani Behr.

    (... and I think the manual names the computer DANI, too. I will check on this one.)
     
  14. AD

    AD Finder of things, Doer of stuff

    Messages:
    4,918
    Likes Received:
    102
    Trophy Points:
    36,885
    Location:
    BC, Canada
  15. HCl

    HCl WC EXE archaeologist

    Messages:
    327
    Likes Received:
    52
    Trophy Points:
    22,935
    Here's another nugget that might be of some interest:D

    As I previously discussed with AD, my recollection was that the RLE-based algorithm used in WC3 and StarLancer for sprite compression might be used in Privateer 2. Today I had some time to go through a few old CDs and found some of my old tools and docs from my StarLancer editing days. Long story short, I found a doc describing this format.

    I did not have the chance to run this algorithm against P2 files, but the StarLancer format seems to start with the "1.40" string, similar to P2, so this at least reinforces my belief that the algorithm is used in there somewhere.

    This is a simple RLE-based compression algorithm with transparency support. If i had to guess, i would say that this would be mostly used on images that require transparency, such as VDU pics, although it is possible things like the Booth could also be compressed with it.

    (old StarLancer VDU image attached, mostly for nostalgia purposes...)

    Anyway, Lin Kuei or someone else interested in P2 editing may want to play with this algorithm a bit. I'm including in the quote block below the doc i wrote back in July 2000, when we were trying to import / export StarLancer VDU pics (wow... it's been 10 years already?)

     

    Attached Files:

  16. Storm

    Storm Spaceman

    Messages:
    22
    Likes Received:
    0
    Trophy Points:
    0
    I haven't bothered with the extraction tools, savegame modifiers, etc., myself... in any case I have a somewhat "moody" Vista machine (my main computer) and there is no guarantee that the utilities would work (unless some of these tools are DOS-compatible).

    For playing P2, I use a legacy Pentium-166 machine, with DOS 6.22 installed (specifically it is a souped-up IBM Personal Computer model 350).

    I thought I might ask / or / provide some focus; what would be some of the long term goals of this work (by those working on it)?

    Suggestions.

    (1) There is at least 1 database entry that is glitched and doesn't occur when it is supposed to, at least in Privateer 17.0e; "The Kindred" (Companies). I SEEM to remember (I THINK) that in Priv2 pre-17.0e, (some patch level, I forget), I THINK that this database entry DOES occur, but apparantly patch 17.0e glitches it. There might be other database entries that are glitched; I think I ran across one or two on the CIC Encyclopedia for the P2 Database, that I had never seen before.

    (2) I kind of hate the way the game "levels up" and progresses through "difficulty levels"; I realize that this is supposed to present increasing challenge; but specifically the increasingly difficult "Ship Trigger per Navpoint" function is a real pain, especially by the time you get to certain situations such as the Bill Maddox escort mission. I HATE being MAROONED at a Nav Point, with a dozen enemies constantly regenerating too quickly.

    (3) I'd like to alter (or have altered) the navpoint characteristics (page 33 in the Official Strategy Guide); chances for faction ships to appear, chances for SOS broadcasts.

    (4) It would be nice if the economy was a little better. I find myself just constantly trading between Hermes (Industrial, Hardware) and Anhur(Ores). It would be nice if there were some additional (more) profitable routes for variety - and it would only be logical.

    (5) Wouldn't it be cool if we could land at the Pirate Bases (Draknor, Kastagan)? AND if we had a reason for doing so? (say, black market items REEL CHEEP). Even better if unique station backgrounds could be done, but if that is impractical, just use the setup for Commodity Stations.
     
  17. AD

    AD Finder of things, Doer of stuff

    Messages:
    4,918
    Likes Received:
    102
    Trophy Points:
    36,885
    Location:
    BC, Canada
    I don't have much to add at the very moment but I *did* find a working download to your SPR2BMP tool here: http://digitality.comyr.com/flyboy/lb2/progs/progs.html. Interestingly he also has the source folder there for download too which has this explanation:
    I'm planning on seeing if I can force it to look at the P2 files. Right now the main problem I see is that the P2 'SPR' chunks are embeded in the IFFs so I think I have to extract just the chunks that start with the ...1.40. strings and then maybe rename them with an SPR extension. I don't know if I would need to recompile with different image dimension info or not.
     
  18. AD

    AD Finder of things, Doer of stuff

    Messages:
    4,918
    Likes Received:
    102
    Trophy Points:
    36,885
    Location:
    BC, Canada
    So, it looks like it doesn't care what the extension is, so that's good. Also, I did make a fraction of progress and can prove that yes there is some relation between p2 and the startlancer files. What I've attempted so far is to either find files where the def chunks unzipped directly into files starting with the 1.40 string, or I looked into a few files that were on the disk not in archives already (such as the two *.SHP files on disk one). Results were... somewhat interesting, but I never managed to extract a good quality picture. Actually, the main error I ran into was that the program would return the number of pictures it thougth was in the 1.40 chunk and then give an astronomically sized estimated picure resolution and then crash.

    [​IMG]

    The exception to this is all the SHP files within Pilots.big and insert.shp within the root of disk one. Insert.shp also crashed the program (as seen in the above pic) but not after finding one picture. I can't tell though if this picture is actually garbled or if it's just lacking a proper palette. Note that you can kind of make out that it says "please insert CD."

    [​IMG]

    This is a similar error to what happens with the pilot.shp files. Those only seem to have one image per SHP file and don't crash the program. This following image should be the pilot VDU pic for Xavier Shondi.
    [​IMG]
     

    Attached Files:

  19. HCl

    HCl WC EXE archaeologist

    Messages:
    327
    Likes Received:
    52
    Trophy Points:
    22,935
    Oh yeah, the Longbow 2 community! Now i remember :) It's been a while since i participated on game modding discussions there. I helped them out sorting some LB2 file formats a few years back, and then noticed they used the same SPR format as StarLancer as well. The spr2bmp program in Flyboy's site is the same one we used for StarLancer with no modifications.

    Looks like it works reasonably well for P2 too, great! The crash reveals an incomplete understanding of the header, which is not surprising since I am skipping some unknown fields there and hoping for the best. Seems like things went reasonably well with StarLancer, but in P2 things are not going so well it seems. Time to look into the header again! :) I'll try a few things out tomorrow, but if anyone finds anything new in the meantime, please post it here.

    The algorithm itself seems to work fine. The images decompress all the way to the end and it seems very reasonable for an image with the wrong palette. I'm convinced that with the right palette the images will look perfect.

    Anyway, nothing else useful from me at this time, but i'll post again later if I think of something.
     
  20. AD

    AD Finder of things, Doer of stuff

    Messages:
    4,918
    Likes Received:
    102
    Trophy Points:
    36,885
    Location:
    BC, Canada
    So I've attached a number ( I left out a few since the max attach number is 10) of the palette images from the P2 source archive (which is mostly texture GIFs and 3DS models). As I think was stated elsewhere P2 doesn't include the palette info in the textures and sprite images themselves. For ship textures the palettes are determined in the model file themselves. In each BMAT chunk of the model IFF (which I assume is a material) there are four chunks which apear to be 1) the material name, 2) specific info about the material 3)which palette to use for the material, and 4) the actual texture image name.

    The palette and texture files are all located in spacetex.iff and all have what looks like a header chunk ahead of the zlib compressed image chunk. However neither the textures nor the palette files use the RLE based compression. I haven't quite figured out what the format is or how to view them. It's unclear to me how exactly the game figures out which palette to use on everything else. I'm thinking that maybe it's defined somewhere that unless otherwise stated it's possible that it uses the texture found in game.iff for everything.... that would be this one: [​IMG]. Or I have to keep looking and there may be a default palette defined elsewhere.
     

    Attached Files:

Share This Page