Privateer 2 editing and extracting

AD

Finder of things, Doer of stuff
So I figured out what I was doing wrong with one of the programs I was using. I was specifying the offset incorrectly. Anyway, do a search for "offset file unzipper" and you'll get the command line program as well as the uncompiled source. This program lets you both unzip each def chunk into a *.dat file which is just the straight code of the chuck or give you the location of the zlib chunks. You can also unpack the zlib chunks into one file.

When I run it on SPACE.IFF I get similar looking code to what HCl posted. Basically you use the '-a' option and specify the file you want, indicate your output directory... and then put in the offset for the first def chunk which is really easy to find using the VIFF tool. When you click on any individual chunk in the list in VIFF you will get a box showing that chunk's code. If the first three letters are DEF then record the offset that is in written in the top of the that particular window, else do a search for DEF in a hex editor.

Usage: [options] <input> <output/dir> <offset>

Options:
-s search for possible zip/gzip data in the input file, the scan starts
from the specified offset and finishs when something is found
the output field is ignored so you can use any name you want
-S as above but continues the scan (just like -a but without extraction)
-a unzip all the possible zip data found in the file. the output
directory where are unzipped the files is identified by <output>
all the output filenames contain the offset where they have been found
-A as above but without unzipping data, the output files will contain the
same original zipped data, just like a simple data dumper
-1 related to -a/-A, generates one unique output file instead of many
-m SIZE lets you to decide the length of the zip block to check if it is a
valid zip data. default is %d. use a higher value to reduce the number
of false positive or a smaller one (eg 16) to see small zip data too
-z NUM this option is needed to specify a windowBits value. If you don't find
zip data in a file (like a classical zip file) try to set it to -15
valid values go from -8 to -15 and from 8 to 15. Default is 15
-q quiet, all the verbose error messages will be suppressed
-r don't remove the invalid uncompressed files generated with -a and -A
Here's a sample of what I got when I ran it on a texture file:
 

Attachments

HCl

WC EXE archaeologist
AD, yep, that looks exactly the kind of hex-patterns i would expect to see on an 8-bit sprite/texture. Looks like that tool you found does the job very nicely as well!

Yep, this would be great. I could then directly convert it into C#, add it to the priv2editor and enhance it this way with a file-editing section.
Alright, i uploaded the Visual Studio C++ project here.

A couple of notes:

- My VS project goes looking for zlib on the "c:\projects\zlib..." directory. This path is absolute, so you will want to go to the project's property pages and change it to your zlib path. Also, and this might be more or less obvious, but when compiling zlib be sure to compile without asm, otherwise you'll have to have Masm installed.

- Since you're developing your tool in C#, i guess you'll need to interface with zlib using either .NET Interop or Managed C++ (the former is probably more straight-forward, but it really depends on how you're doing things on your end). Also, my project links zlib statically, but I am betting it will probably be easier for you to link it dynamically.

Edit: actually, after hitting "submit", it occurred to me that there should already be a port of zlib for .NET out there. Sure enough, there is one, so you may want to take a look at it: http://www.componentace.com/zlib_.NET.htm
 

AD

Finder of things, Doer of stuff
So I took a look at MUSIC.IFF. I can say that the "offset file unzipper" works great but that I didn't really learn anything new. MUSIC.IFF has... a couple sections. THere's a BMSQ section with six BMSD subsections that would at a guess correspond to the six song files... possibly instrument or usage or soundfont info perhaps? That's followed by the six zlib compressed song files themselves in the BMTN section. Running the zlib unpacker extracts all six song files. If you used the program I mentioned then all you have to do is rename the file extension to XMI.

The P2 ingame music is all XMI files which were some kind of proprietary extended MIDI format that can be a pain to find programs to play. Awave studio does it but it's one of those timed trial deals. There's probably a winamp pluggin out there or something though. Someone obviously extracted these one upon a time though as the files I ended up with are identical to the ones in the CICs files section (There's a zip with both the XMI's and versions converted to regular MIDI).

I guess it would be possible to replace the inflight music with custom midi's but I would bet people would rather work out a way to have custom playlists of MP3s or something which is probably a lot harder to implement. On the otherhand, MIDI support in newer versions of windows is dwindling.
 

Wedge009

Rogue Leader
I find the MIDI plug-in included with the standard Winamp package is usually pretty good at playing even the most esoteric MIDI formats.
 

AD

Finder of things, Doer of stuff
I find the MIDI plug-in included with the standard Winamp package is usually pretty good at playing even the most esoteric MIDI formats.
Possibly. I found stuff suggesting winamp plays XMI files, yet couldn't find it in the latest version, though I refuse to pay for the pro version and it's possible they moved XMI support to the pro version. I really didn't look that hard for a plugin either.
 
Thank you HCl. As soon as i've find some time i'll give it a test and start creating a user friendly editor for the files.
Also thanks for finding the .net zlib library.

@AD: I just found this: xmi2mid

a quote from a different forum (here)
Are you trying to extract old game sound files or something?

WinAmp can convert XMI files to midi. Give it a shot.

Under Options/Preferences choose Plugins/Input then click "Nullsoft MIDI player", then click Configure.

Select "File Types", select all file types and then click OK.

Winamp should now play XMI files. Open one up and click File/View file info.

From there you can save it as a Midi file.
 

AD

Finder of things, Doer of stuff
@AD: I just found this: xmi2mid

a quote from a different forum (here)
Cool! That worked, thanks. The only other thing I had to do was in the midi configuration switch volume control from auto to driver-specific. Otherwise it was doing some funky things to the sound that I can't quite explain.
 

Wedge009

Rogue Leader
If you found something that works for you, then good, but I assure you that you don't need to pay for Winamp to get the MIDI plug-in to play XMIs. I've used it to convert and/or play extracted MIDI data from fun DOS-era games like Syndicate and Lemmings Chronicles.
 

Bob McDob

Better Health Through Less Flavor
I find the MIDI plug-in included with the standard Winamp package is usually pretty good at playing even the most esoteric MIDI formats.
Well, it plays, all right; but I can't say that the data that comes out sounds much like the music in-game.
 

Wedge009

Rogue Leader
It's dependent on what output format you're using, of course, as well as what MIDI-capabilities your hardware has available, if any.
 

HCl

WC EXE archaeologist
Speaking of which, i extracted some MIDIs back in the day as well, they're still in my site here.

It's probably the same XMIs you just extracted, I remember i also used a similar xmi2mid tool to convert the files to a more supported format. The only thing noteworthy in my mind is that i extracted these before i figured out the Def! chunk compression, using a small asm utility for creatively dumping memory from the game at given points. For this reason, there might be a few midis that I missed.
 

AD

Finder of things, Doer of stuff
Yes, the fat files are definitely the priv2 audio format.

Iused the bigf extractor tool that i've found somewhere here for download, to extract some files from the privateer 2 archives. Among others also the sound files from dani.big. With audacity's import raw data using "VOX ADPCM" and a sample rate of 11025Hz they where hearable almost 100% like ingame, but still had some strange echo effects at the end and some rare clicks/disturbances. Does anyone know the correct settings or has written a tool to import correct the audio files? It could be also nice to have these sounds available to hear in WCPedia.
Did some experimenting with this. The samples in speech.big definitely sound best at 11025. I can't tell if the extra noise is on purpose but the files do sound noisy within the game. However, I found that the ones in DANI.BIG sounded best at 16000. DANI... not sure why it's called that but it seems to be the all the ship's computer audio.

So, I was looking at some of the IFFs contained within the isets and sets IFF files. Curiously a large chunk of the ISETS files looked an awful lot like the FAT files so I ran them through a similar process... and wouldn't you know? The isets files are almost strictly the FMV choice dialogue. They played very clearly at 22050 khz. Looking at anhur_tf.IFF you can see four separate 'speech' chunks with associated 'text' chunks. This is somewhat corroborated by the audio itself which is basically four sets of choice dialogue (8 lines, 4 pairs)

Well why not? Money's money.
Nah, I don't like this political stuff.

Taffin! Well, Ok, why not let him think that?
There's no point in lying to this guy.

Sure, Why not?
Sounds a little fishy.

Sounds like a good deal.
No, I don't think so.
Now there's definitely also audio within the SETS.iff files as well but it's a much smaller portion of the file itself. Most of the file just plays like garbage but there's a distinct section at the start that I'm pretty sure are the background noises you hear when you are on a planet at... say, the customs screen. Some of the effects sounded a bit garbled at 22050 and others like the crowd chatter sounded very clear. Looking at something like anhur.iff I would guess the effects are in the very beginning FATF chunk. My guess would be that the rest of the file is hotspot info and animation frames for the various base 'objects'.
 

Bandit LOAF

Long Live the Confederation!
DANI... not sure why it's called that but it seems to be the all the ship's computer audio.
The Bitching Betty voice was voiced by actress Dani Behr.

(... and I think the manual names the computer DANI, too. I will check on this one.)
 

HCl

WC EXE archaeologist
Here's another nugget that might be of some interest:D

As I previously discussed with AD, my recollection was that the RLE-based algorithm used in WC3 and StarLancer for sprite compression might be used in Privateer 2. Today I had some time to go through a few old CDs and found some of my old tools and docs from my StarLancer editing days. Long story short, I found a doc describing this format.

I did not have the chance to run this algorithm against P2 files, but the StarLancer format seems to start with the "1.40" string, similar to P2, so this at least reinforces my belief that the algorithm is used in there somewhere.

This is a simple RLE-based compression algorithm with transparency support. If i had to guess, i would say that this would be mostly used on images that require transparency, such as VDU pics, although it is possible things like the Booth could also be compressed with it.

(old StarLancer VDU image attached, mostly for nostalgia purposes...)

Anyway, Lin Kuei or someone else interested in P2 editing may want to play with this algorithm a bit. I'm including in the quote block below the doc i wrote back in July 2000, when we were trying to import / export StarLancer VDU pics (wow... it's been 10 years already?)

The file starts with a 4 byte string: "1.40". This is assumed to be the file version, which is why i was surprised the unpacking routine worked: WC3 files were version "1.11". You'll find on offset 4 a dword (4 bytes), which indicates the number of images stored in this SPR file. Afterwards, you'll find what seem to be 8-byte offsets that indicate where in the file each image starts.

Image data holds (most of the times, although not always) image information, such as image dimensions. More specifically:

37 00 47 00 01 00 01 00 09 00 00 00 01 00 00 00
35 00 00 00 2D 00 00 00 01 1B 09 BE 04 55 67 00
01 19 11 67 29 29 04 29 29 BE BE 00 01 18 15 10

The first 8 bytes are unknown and are being ignored right now. 8 bytes after the start of the block, you'll find 2 dwords indicating where the image should be drawn (in this case 9 pixels to the right and 1 pixel down). Then you'll find 2 more dwords which specifies the image dimensions, but you'll have to add 1 to their value (so in this case the image size is 54x46). The rest is compressed data that'll have to be passed to the unpacker.

After the data is unpacked, you'll have an image with an 8-bit color depth, so you'll have to load a palette so it'll look good. As far as ship targetting images go: software versions (soft_*.spr) use the palette stored in softpal.tga. The hardware ones use a palette that wasn't found in any TGA file, so thanks go to Dustin Evans, who was able to come up with a palette based on screenshots and on the software palette (Dustin's palette is the one used on BMPs created by spr2bmp).

A small detail: the dimension of the image you have to unpack is not 54x46, which would be logic. Instead, it is 54-9 x 46-1 (because you're not drawing a rectangle starting at 0,0 , it's starts at 9,1). In the spr2bmp utility, i made this subtraction and the image fits perfectly on these dimensions. It's important to understand this since you may have to play with these 4 values if the image doesn't appear in the game exactly where you'd want it to be.



The unpacking algorithm:

The algorithm unpacks the image a row at a time, so you'll have to keep a counter with the number of rows to unpack (there are some more variables you'll need to have, but they should become obvious as the algorithm is described).
Assume that it begins executing in "Begin" and finishes working in "End":

- Begin
- Read a byte
- Remove the less significative bit (LSB) from the read byte, but keep it in memory
- If LSB is zero
- If the rest of the byte is 0, this row has no more bytes. If there aren't any more rows to unpack, End. Otherwise, increase row counter (and adjust the row offset) and go to Begin.
- If the rest of the byte is not 0, use the value of the byte currently in memory as a counter (x). Read one more byte, write it x times and go to Begin.
- If LSB is one
- If the rest of the byte is 0, read one more byte, add it's value to the output pointer (skipping a given number of bytes) and go to Begin.
- If the rest of the byte is not 0, use the value of the byte currently in memory as a counter (x). Move x bytes from the "compressed" zone of memory to the "uncompressed" zone.


That's pretty much it. I'm including the source code i created that implements it:


-----------------------8<------------------------------------

// value subtraction on xDim and yDim done before calling
void _unpack(BYTE * in, BYTE * out, DWORD xDim, DWORD yDim)
{
BYTE b;
int inPtr=0, outPtr=0, i, rowCounter = 0, counterX;
char lsb = false;

while (1)
{
// read byte
b = in[inPtr++];
lsb = (b & 1);
b >>= 1;

if (!lsb) // if lsb == 0
{
if (!b) // if lsb == 0 and byte == 0
{
// increase row counter
rowCounter ++;

// setup correct row offset
outPtr = rowCounter * xDim;

if (rowCounter >= yDim) // are we done?
return;
}

else // if lsb == 0 and byte != 0
{
// setup counter
counterX = (int) b;

// read another byte
b = in[inPtr++];

// write the byte X times
for (i=0; i<counterX; i++)
out[outPtr++] = b;

continue;
}

}

else // if lsb == 1
{
if (!b) // if lsb == 1 and byte == 0
{
// read a new byte and add it to outPtr
outPtr += (int) in[inPtr++];

continue;
}

else // if lsb == 1 and byte != 0
{
// setup counter
counterX = (int) b;

// copy string
for (i=0; i<counterX; i++, inPtr++, outPtr++)
out[outPtr] = in[inPtr];

continue;
}

}

}

}

-----------------------8<------------------------------------
 

Attachments

Storm

Spaceman
I haven't bothered with the extraction tools, savegame modifiers, etc., myself... in any case I have a somewhat "moody" Vista machine (my main computer) and there is no guarantee that the utilities would work (unless some of these tools are DOS-compatible).

For playing P2, I use a legacy Pentium-166 machine, with DOS 6.22 installed (specifically it is a souped-up IBM Personal Computer model 350).

I thought I might ask / or / provide some focus; what would be some of the long term goals of this work (by those working on it)?

Suggestions.

(1) There is at least 1 database entry that is glitched and doesn't occur when it is supposed to, at least in Privateer 17.0e; "The Kindred" (Companies). I SEEM to remember (I THINK) that in Priv2 pre-17.0e, (some patch level, I forget), I THINK that this database entry DOES occur, but apparantly patch 17.0e glitches it. There might be other database entries that are glitched; I think I ran across one or two on the CIC Encyclopedia for the P2 Database, that I had never seen before.

(2) I kind of hate the way the game "levels up" and progresses through "difficulty levels"; I realize that this is supposed to present increasing challenge; but specifically the increasingly difficult "Ship Trigger per Navpoint" function is a real pain, especially by the time you get to certain situations such as the Bill Maddox escort mission. I HATE being MAROONED at a Nav Point, with a dozen enemies constantly regenerating too quickly.

(3) I'd like to alter (or have altered) the navpoint characteristics (page 33 in the Official Strategy Guide); chances for faction ships to appear, chances for SOS broadcasts.

(4) It would be nice if the economy was a little better. I find myself just constantly trading between Hermes (Industrial, Hardware) and Anhur(Ores). It would be nice if there were some additional (more) profitable routes for variety - and it would only be logical.

(5) Wouldn't it be cool if we could land at the Pirate Bases (Draknor, Kastagan)? AND if we had a reason for doing so? (say, black market items REEL CHEEP). Even better if unique station backgrounds could be done, but if that is impractical, just use the setup for Commodity Stations.
 

AD

Finder of things, Doer of stuff
As I previously discussed with AD, my recollection was that the RLE-based algorithm used in WC3 and StarLancer for sprite compression might be used in Privateer 2. Today I had some time to go through a few old CDs and found some of my old tools and docs from my StarLancer editing days. Long story short, I found a doc describing this format.

I did not have the chance to run this algorithm against P2 files, but the StarLancer format seems to start with the "1.40" string, similar to P2, so this at least reinforces my belief that the algorithm is used in there somewhere.

This is a simple RLE-based compression algorithm with transparency support. If i had to guess, i would say that this would be mostly used on images that require transparency, such as VDU pics, although it is possible things like the Booth could also be compressed with it.

(old StarLancer VDU image attached, mostly for nostalgia purposes...)

Anyway, Lin Kuei or someone else interested in P2 editing may want to play with this algorithm a bit. I'm including in the quote block below the doc i wrote back in July 2000, when we were trying to import / export StarLancer VDU pics (wow... it's been 10 years already?)
I don't have much to add at the very moment but I *did* find a working download to your SPR2BMP tool here: http://digitality.comyr.com/flyboy/lb2/progs/progs.html. Interestingly he also has the source folder there for download too which has this explanation:
This program will convert Longbow 2 SHP files to BMP. It originally was created
for Wing Commander and StarLancer SPR files, but it seems to work on Longbow 2
perfectly (pending palette change).
I'm planning on seeing if I can force it to look at the P2 files. Right now the main problem I see is that the P2 'SPR' chunks are embeded in the IFFs so I think I have to extract just the chunks that start with the ...1.40. strings and then maybe rename them with an SPR extension. I don't know if I would need to recompile with different image dimension info or not.
 

AD

Finder of things, Doer of stuff
I don't have much to add at the very moment but I *did* find a working download to your SPR2BMP tool here: http://digitality.comyr.com/flyboy/lb2/progs/progs.html. Interestingly he also has the source folder there for download too which has this explanation:


I'm planning on seeing if I can force it to look at the P2 files. Right now the main problem I see is that the P2 'SPR' chunks are embeded in the IFFs so I think I have to extract just the chunks that start with the ...1.40. strings and then maybe rename them with an SPR extension. I don't know if I would need to recompile with different image dimension info or not.
So, it looks like it doesn't care what the extension is, so that's good. Also, I did make a fraction of progress and can prove that yes there is some relation between p2 and the startlancer files. What I've attempted so far is to either find files where the def chunks unzipped directly into files starting with the 1.40 string, or I looked into a few files that were on the disk not in archives already (such as the two *.SHP files on disk one). Results were... somewhat interesting, but I never managed to extract a good quality picture. Actually, the main error I ran into was that the program would return the number of pictures it thougth was in the 1.40 chunk and then give an astronomically sized estimated picure resolution and then crash.



The exception to this is all the SHP files within Pilots.big and insert.shp within the root of disk one. Insert.shp also crashed the program (as seen in the above pic) but not after finding one picture. I can't tell though if this picture is actually garbled or if it's just lacking a proper palette. Note that you can kind of make out that it says "please insert CD."



This is a similar error to what happens with the pilot.shp files. Those only seem to have one image per SHP file and don't crash the program. This following image should be the pilot VDU pic for Xavier Shondi.
 

Attachments

HCl

WC EXE archaeologist
Oh yeah, the Longbow 2 community! Now i remember :) It's been a while since i participated on game modding discussions there. I helped them out sorting some LB2 file formats a few years back, and then noticed they used the same SPR format as StarLancer as well. The spr2bmp program in Flyboy's site is the same one we used for StarLancer with no modifications.

Looks like it works reasonably well for P2 too, great! The crash reveals an incomplete understanding of the header, which is not surprising since I am skipping some unknown fields there and hoping for the best. Seems like things went reasonably well with StarLancer, but in P2 things are not going so well it seems. Time to look into the header again! :) I'll try a few things out tomorrow, but if anyone finds anything new in the meantime, please post it here.

The algorithm itself seems to work fine. The images decompress all the way to the end and it seems very reasonable for an image with the wrong palette. I'm convinced that with the right palette the images will look perfect.

Anyway, nothing else useful from me at this time, but i'll post again later if I think of something.
 

AD

Finder of things, Doer of stuff
So I've attached a number ( I left out a few since the max attach number is 10) of the palette images from the P2 source archive (which is mostly texture GIFs and 3DS models). As I think was stated elsewhere P2 doesn't include the palette info in the textures and sprite images themselves. For ship textures the palettes are determined in the model file themselves. In each BMAT chunk of the model IFF (which I assume is a material) there are four chunks which apear to be 1) the material name, 2) specific info about the material 3)which palette to use for the material, and 4) the actual texture image name.

The palette and texture files are all located in spacetex.iff and all have what looks like a header chunk ahead of the zlib compressed image chunk. However neither the textures nor the palette files use the RLE based compression. I haven't quite figured out what the format is or how to view them. It's unclear to me how exactly the game figures out which palette to use on everything else. I'm thinking that maybe it's defined somewhere that unless otherwise stated it's possible that it uses the texture found in game.iff for everything.... that would be this one:
. Or I have to keep looking and there may be a default palette defined elsewhere.
 

Attachments

Top