Privateer 2 editing and extracting

[Lin Kuei Ominae]

You don't have to jump to the nearest planet. When you load a savegame you are already on that planet that has these set/modified bbs cards. So just enter the booth, pick the mission and start from the planet to play it.
However, i still haven't figured out the corresponding mission to a certain mission number.
e.g. there are Missions with the index number 368, while the missions 369 and 367 don't exist. This seems to be more like a pointer to a certain mission rather than an enumeration of one mission after another (e.g. like read mission at line 368 in the mission*.iff file, where only line 368 has an entry point and the following line has already certain data to this mission and the previous line 367 data to the previous mission). So it can still take a bit time to find the really working mission indexes.
Right now i'm working on a way to store the data user-friendly and allow the user to expand/modify this file (most probably an ini with sections like [368] Name="Ser Arl Shoyer's ass kicked" Reward="5.500Credits").

I'm still searching the news hex address. Like the string "BBS_", which is followed by the section of the bbs cards or "PCOM", which is followed with the planet commodities, there is also a special "NEWS" string in the savegame file.
However unlike the other sections, this NEWS section doesn't seem to contain the information about the currently shown news.
But i haven't given up on that and will surely sooner or later find the place where these information are stored. A certain tab in the editor is already included, and this will then also allow to modify all related data to the news.

After that is done, there also isn't much left that needs to be deciphered from the savegame, as only the big section with the accepted missions is left and this one looks really complicated due to the many dynamic variables like ships involved in the mission, their nav point, relative position from the nav points centre, direction speed, interaction with other ships etc etc.
I'm not sure if i ever implement this, but this will surely go in the last version of the editor. Though i also know it would be very useful to directly read all the corresponding data concerning a certain mission, so you could quickly compare the WCPedia mission page with the real mission data. Then it might be even possible to write complete new missions and hook them into the game via editing the savegame. But as i said, this is the very last point on my list.
I'm much, much more interested to know how i can modify the ship speed, shield, hull, yaw/pitch/roll values as well as the weapon firepower range etc. Though this data doesn't seem to be stored in the dark.exe, but in one of the strangely compressed iff files (i assume gameflow.iff).

 

[AD]

You don't have to jump to the nearest planet. When you load a savegame you are already on that planet that has these set/modified bbs cards. So just enter the booth, pick the mission and start from the planet to play it.
However, i still haven't figured out the corresponding mission to a certain mission number.
e.g. there are Missions with the index number 368, while missions with 369 and 367 don't exist. This seems to be more like a pointer to a certain mission rather than an enumeration of one mission after another (e.g. like read mission at line 368 in the mission*.iff file, where only line 368 has an entry point and the following line has already certain data to this mission and the previous line 367 data to the previous mission). So it can still take a bit time to find the really working mission indexes.
Right now i'm working on a way to store the data user-friendly and allow the user to expand/modify this file (most probably an ini with sections like [368] Name="Ser Arl Shoyer's ass kicked" Reward="5.500Credits").

I'm still searching the news hex address. Like the string "BBS_", which is followed by the section of the bbs cards or "PCOM", which is followed with the planet commodities, there is also a special "NEWS" string in the savegame file.
However unlike the other sections, this NEWS section doesn't seem to contain the information about the currently shown news.
But i haven't given up on that and will surely sooner or later find the place where these information are stored. A certain tab in the editor is already included, and this will then also allow to modify all related data to the news.

After that is done, there also isn't much left that needs to be deciphered from the savegame, as only the big section with the accepted missions is left and this one looks really complicated due to the many dynamic variables like ships involved in the mission, their nav point, relative position from the nav points centre, direction speed, interaction with other ships etc etc.
I'm not sure if i ever implement this, but this will surely go in the last version of the editor. Though i also know it would be very useful to directly read all the corresponding data concerning a certain mission, so you could quickly compare the WCPedia mission page with the real mission data. Then it might be even possible to write complete new missions and hook them into the game via editing the savegame. But as i said, this is the very last point on my list.
I'm much, much more interested to know how i can modify the ship speed, shield, hull, yaw/pitch/roll values as well as the weapon firepower range etc. Though this data doesn't seem to be stored in the dark.exe, but in one of the strangely compressed iff files (i assume gameflow.iff).

Don't know if it's relevant or not but the WC prophecy dev cd stuff Cpt Johnny sent had a program for viewing IFF files that would organize all the data for you. Not sure that it works with P2 but it was interesting for WC3, 4, and even the PSX files. (called VIFF.exe IIRC). So as long as you can extract the archived IFFs off the P2 cds it might give you a good tool for hunting for stuff in the hex editor.

 

[Lin Kuei Ominae]

Can you give me a link to this VIFF tool? Mario Brito wrote in the comments of his bigf privateer 2 extractor tool that the compression of privateer 2 and wingcommander is in some cases very similar but in others totally incompatible/different (e.g. zlib for some of the priv2 files).
Maybe the tool works on the right files and i can uncompress them to finally read/write the ship data.

 

[AD]

Can you give me a link to this VIFF tool? Mario Brito wrote in the comments of his bigf privateer 2 extractor tool that the compression of privateer 2 and wingcommander is in some cases very similar but in others totally incompatible/different (e.g. zlib for some of the priv2 files).
Maybe the tool works on the right files and i can uncompress them to finally read/write the ship data.

Unless someone beats me to it I'll have look for the link when I get back from work. You probably need to extract the IFFs first for sure though.

 

[AD]

This should be a link to the dev CD downloads: https://www.wcnews.com/news/update/8858

It's a bit of a bigger download but I can probably just post the VIFF program later if there's a need. I recommend not setting compatabilty on VIFF95.exe and dragging the IFFs into it instead of doubleclicking the iffs to open. Not sure how useful it will be but here's a shot of what you see when you load P2s booth.iff:

When you load an IFF you get the breakdown list on the left that gives you headers and also the location. doubleclicking a chunck in the list gives you the Hexedit-like box containing the code for that specific chunk.

 

[Lin Kuei Ominae]

This should be a link to the dev CD downloads: https://www.wcnews.com/news/update/8858

I checked out the VIFF.exe (thanks for the download link btw), but unfortunately it didn't help me further. Because it can't uncompress the chunks (neither extracting them) and i also haven't found a converter to convert the chunks into a common picture or readable text format.

 

[AD]

I checked out the VIFF.exe (thanks for the download link btw), but unfortunately it didn't help me further. Because it can't uncompress the chunks (neither extracting them) and i also haven't found a converter to convert the chunks into a common picture or readable text format.

Yeah, it's not so much a decompression tool as it organizes the IFF file (and other files too) by chunk and gives you the addresses of each individual chunck. As far as I can tell the only files where the BIGF tool is useful are SETS.IFF, ISETS.IFF and PILOTS.BIG and SPEECH.BIG and DANI.BIG. When I compiled HCLs tool it was useful to extract a bunch of *.shp files *.FAT and *.fat files and a bunch of IFFs for planets and stuff. Presumably since they were in speech.big many of the *fat files are audio.

As far as what compression is used beyond that, I wouldn't be the one to ask. Oddly though SAMPLE.OPL contains a list of all the files in DANI.BIG.

 

[Lin Kuei Ominae]

Yeah, it's not so much a decompression tool as it organizes the IFF file (and other files too) by chunk and gives you the addresses of each individual chunck. As far as I can tell the only files where the BIGF tool is useful are SETS.IFF, ISETS.IFF and PILOTS.BIG and SPEECH.BIG and DANI.BIG. When I compiled HCLs tool it was useful to extract a bunch of *.shp files *.FAT and *.fat files and a bunch of IFFs for planets and stuff. Presumably since they were in speech.big many of the *fat files are audio.

As far as what compression is used beyond that, I wouldn't be the one to ask. Oddly though SAMPLE.OPL contains a list of all the files in DANI.BIG.

Yes, the fat files are definitely the priv2 audio format.
In one of my posts above, i did something similar and found a quite decent solution to convert the fat audio files.
let me quote myself.

Oh btw, yesterday i used the bigf extractor tool that i've found somewhere here for download, to extract some files from the privateer 2 archives. Among others also the sound files from dani.big. With audacity's import raw data using "VOX ADPCM" and a sample rate of 11025Hz they where hearable almost 100% like ingame, but still had some strange echo effects at the end and some rare clicks/disturbances. Does anyone know the correct settings or has written a tool to import correct the audio files? It could be also nice to have these sounds available to hear in WCPedia.

 

[AD]

Yes, the fat files are definitely the priv2 audio format.
In one of my posts above, i did something similar and found a quite decent solution to convert the fat audio files.
let me quote myself.

It probably doesn't tell you anything you don't already know but further looking into the files spacegeo.iff and spacetex.iff it appears that these contain geometry and textures respectively. The IFFs within spacegeo all start with "FORM BR3D" and have "MAT" sections that each have a defined texture listed. Although the listed textures say *.pix (which it seems is an Alias Wavefront format) they perfectly match the names of all the IFFs in spacetex.iff

In the geometry files the textures are listed in the last chunk (MCMP) and it would seem the palette is defined in the second to last chunk of each MAT section (MSNM) which matches the assumptions Gremule made from looking at some of the source files a while back (https://www.wcnews.com/news/update/10376;https://www.wcnews.com/news/update/9629).

Textures in Privateer 2 are stored without palette info, appropriate palette is chosen on-the-fly. My guess is those gifs in TABS.GIF are those palettes although gifs themselves use standard game palette. You can see by the names, for example CV-BL-GY.GIF means blue->gray.

In the IFFs for the... meshes I guess we could call it rather than geometry, the palate is listed as a .tab which corresponds to that Tab.gif Gremule mentions. So for example can1_d1.iff uses the game.tab palette for each of it's five texture files. Comm_d1 however lists 11 mat sections (therefore eleven textures) and 'cv.bl.br.tab' as the palette for each texture.

 

[Lin Kuei Ominae]

yep. They even have the same filenames as in the P2_Archive.zip, which has the geometry in 3ds and the textures as gif.
So an experience decoder should be actually able to compare the files and find the correct compression algorithm, to uncompress/decode the used iff files.

Damn, it would be awesome to be able to change some 3d-models. The first thing i would do is giving the stream laser MKII a small dual laser beam and the Kraven MKIV a quad laser beam (or big dual laser beam), so the laser beams match with the weapon 3d-model shown in the booth. (i always wondered how the Kraven MKIV can fire a triple laser beam while the gun has only 2 big and 2 small barrels)
As a next step i would make all the laser a lot more different, so they aren't only simple lines with the same length but different texture, but also a sphere (e.g. for the volt laser) and different in size and length. (This was definitely a great advantage of Privateer 1 where each weapon also had a different unique looking projectile)

 

[Bob McDob]

I forgot to post this.

http://reckoning.solsector.net/Privateer2Ever/P2Tex.zip

Using these palettes and a copy of Adobe Photoshop CS, I manually shifted the colors on the textures. Put the models from the archive into one of the folders and it should show up in the proper colors, assuming that the texture exists - I didn't do every color for every GIF.

 

[AD]

Ok, I've split off the Privateer 2 hacking stuff into it's own thread out of the WCpedia discussion since this particular tangent has been taking it's own course and gettinng interesting in it's own right.

Lin Kuei, I did email HCl about his bigf extractor tool. near as I can tell he never actually compiled a version for others to use that included zlib. I've been attempting to add zlib into it myself but honestly - as you may have guessed - I pretty much have no idea what I'm doing. I'm fumbling in the dark. Anyway he may have some more stuff stored away but I got the impression that it would take a while for him to come up with anything, but if you have a specific question I can always forward that to him.

Anyway, here's the basics of what HCl said he could remember about his exploration in to the P2 formats:

You may want to tweak my tool to decompress the Def! chunks with zlib. I recall some chunks were compressed, so you could access some more data with that improvement. I also seem to recall (but don't hold me to it) that P2 uses the same sprite format as WC3/4 PC and StarLancer. If so, it may be worthwhile to take a look at that (it's a simple RLE routine, with transparency support... in a way, not very different from WC1/2 and WCP's RLE.

Also, i did some work on extracting the textures and meshes of P2 models. I don't recall if i ever uploaded anything into my site at some point. Probably not... The only thing that remains there is this pic, from a small experiment i made with the Java3D API and P2 game meshes (the textures are wrong, i ended up using some generic rock texture, for some strange reason...)

He're's first HCls text out of the bigf source code itself and then a follow-up discussion we had:

Privateer2 uses the 'zlib' Library for uncompressing data, which is
* Copyright (C) 1995-1996 Jean-loup Gailly and Mark Adler.
* Fortunately the authors allow it to be used freely. Feel free to complete
* this source code by linking Jean and Mark's routines, so that the
* extracted IFFs will not have any compressed chunks. Be sure to send
* me a copy tho I frankly don't know if P2 will accept uncompressed
* chunks but it'll be 100% human-readable that way (ie, hex-edit).
* Also, since the authors also supply the respective compression
* routines with the zlib package, we can compress chunks.
* P2 editing can be very interesting now A separate chunk unpacker
* is available, but i'm sure it'll become obsolete very soon.
*
* zlib isn't the only compression routine used in P2: a routine
* similar to WC3 Routine1 is used in P2 (as a matter of fact, WC3 R1
* partially unpacked a P2 chunk). I'm calling this routine P2
* Extract1 (P2 E1, for short). Not that it matters much what i call
* the routines, but i have to have a way to tell them apart. It's
* clear now that most WC games have more than one compression routine
* (i tend to oversimplify things, so i didn't really consider that
* possibility before working on MGIs.. but i'm rambling now). I also
* found a 3rd compression routine (which is used for text mostly).
* The information area will be uploaded with this when possible.

There’s also a short blurb in the chunk tool as follows:

/* After reading the file, i'll do a simple search for the "Def!" string
* (which signals the compression) and decompress it into a separate file.
* chunkCounter is used for the name of the files. In the near future
* chunk decompress will probably take place on the P2 IFF extractor itself.

- The "Def!" chunks are compressed, as the name suggests, with the Deflate algorithm. As I recall, zlib has a deflate function, so that's what you want to call. I *think* deflate takes a size parameter, basically in order to tell the algorithm when to stop decompressing. This can either be the size of the chunk itself or an integer (dword) stored on the chunk itself... this detail i don't quite recall. If i had to guess, i would say the decompressed size is stored inside the chunk itself, before the start of the compressed data stream (it's one of those decisions that just would "make sense", given that the last byte of the compressed bitstream will have some leftover bits... so you'd need to either store the decompressed length or use an "end of stream" symbol in your algorithm).

- Yep, there were a few extra decompression algorithms there, but much simpler in structure than whatever you can find on zlib. I think one of them was used for text, so Lin Kuei may have already have this one sorted out. This may turn out not to be an issue for you. (of course, the easiest route would probably be getting Paul Hughes to give you whatever text is compressed, if he still has it).

Also, in order to decompress those chunks you will probably need to have some kind of routines to handle IFF files, since you will get more data than can fit on the chunk. Of course, it is probably simple to generate a text representation of the IFF and feed it into Thomas Bruckner's IFF compiler.

Now, I might continue to try and figure out how to include the Zlib stuff but I'm certain someone with actual programming skills will have a lot more success than me. (They'll definitely get it done a lot faster than me at least) However of use I also managed to fine a command line tool for un-zlibing raw code chunks. That might be useful if I can figure out how it works.

 

[Lin Kuei Ominae]

if you have a specific question I can always forward that to him.

I would like to know how he found out the algorithm just out of interest (a simple trial and error would surely have taken ages).
Though beside of that a code example to extract every file format would be really great.
I would then enhance my Priv2Editor with a File-modding section that gives the normal user the chance to edit/compress/uncompress the p2d files with a graphical interface and a simple file browser.

If that is too much work, it would be awesome if he could point me to the file with all the ship data like shield strength, speed, armor, gun/module/missile slots etc as this doesn't seem to be stored inside the dark.exe.

Other than that every tiny bit of information he can provide is useful.

Maybe also point him to this topic so we can communicate directly.

 

[AD]

I would like to know how he found out the algorithm just out of interest (a simple trial and error would surely have taken ages).

Though beside of that a code example to extract every file format would be really great.

I'll definitely be emailing him again, and he did mention that he might have some older resources stored somewhere so I'll see what he says and definitely mention that I started this thread ( I had pointed him previously to the other thread).

With regards to samples etc, all I can say is don't get your hopes too high. HCl has had a lot on his plate personally and hasn't been able to dedicate much time to this sort of thing lately. I'm not sure much will change in that regard but it's definitely something he's interested in and that's near to his heart.

 

[Lin Kuei Ominae]

Well if he can point me in the right direction (code snippet), i think i can come up with the rest. It's just that i have absolutely no clue how he found out the right hex values for e.g. table1 in his Priv2 TGV audio decompressor or some of the other algorithms operating principles.

 

[Bob McDob]

It'd be great if we could rip models from the game - I think I pointed out in the other thread that the Chirichan Demon model from Paul Hughes' pack is corrupt.

 

[AD]

Well if he can point me in the right direction (code snippet), i think i can come up with the rest. It's just that i have absolutely no clue how he found out the right hex values for e.g. table1 in his Priv2 TGV audio decompressor or some of the other algorithms operating principles.

So I mentioned your question to HCl and here's the quick answer:

I disassemble the EXE files in order to discover the algorithms, basically. My procedure right now involves using a disassembler, a hex editor and a few quickly hacked-together Matlab scripts for testing ideas. Once i get a basic understanding of how things work, i can then re-implement the disassembled code in C, which usually materializes into a command-line utility that i release for the community (eg, my small SegaRIP tool).

Anyway he mentioned that he'll try and look through his old stuff and see if he can remember more and possibly pop in and let us know what he finds here.

 

[Lin Kuei Ominae]

I see, another disassembling expert Coming from the C&C modding community, i'm quite familiar with this kind of elite modders.

Thanks AD for forwarding my question and thanks HCl for the answer.

I'm eager so get more info on the various P2 filetypes.

 

[HCl]

Hey all!

Sorry for not keeping an eye on the Forums as often as i should, i'll make an effort to keep up...

Lin Kuei, love your editing tool, great work!

Due to past e-mail exchanges with AD i took another (very quick) look at the Def! chunks in P2. I made a quick program to test things out and it turns out most of what i recalled was correct: Def! chunks are indeed compressed with zlib's deflate, which means you only have to call the inflate function to get the data back. The compressed chunks have a "Def!" identifier, followed by the uncompressed length of the data block, compressed length, and finally the deflated compressed stream.

This means we have all we need to decompress the stream right there! Good

So i basically came up with this quick program. It decompresses the first Def! chunk on SPACE.IFF (lots of stuff hardcoded, this is not a complete tool)

(quoting most relevant sections only.. still quite large, sorry about that)

#include <zlib.h>

typedef struct _DefHeader
{
unsigned int id;
unsigned int uncomprLen;
unsigned int comprLen;
} DefHeader;


// ---- setup these
Byte * compr = NULL;
Byte * uncompr = NULL;
uLong comprLen;
uLong uncomprLen;
int offset;


// In the P2 Demo at least, the Def! chunk we want starts at offset 0x20 in space.iff...
// We hardcode these values for testing purposes and push it into zlib's inflate
offset = 0x20;
fseek(f, offset, SEEK_SET);
fread(&header, sizeof(header), 1, f);

comprLen = header.comprLen;
uncomprLen = header.uncomprLen;
compr = (Byte *) malloc(comprLen);
uncompr = (Byte *) malloc(uncomprLen);

fread(compr, comprLen, 1, f);
// -----------


int ret;
z_stream d_stream;

d_stream.zalloc = (alloc_func)0;
d_stream.zfree = (free_func)0;
d_stream.opaque = (voidpf)0;

d_stream.next_in = compr;
//d_stream.avail_in = 0;
d_stream.next_out = uncompr;

d_stream.avail_in = comprLen;
d_stream.avail_out = uncomprLen;

printf ("Initializing deflate stream... ");
ret = inflateInit(&d_stream);
printf ("(ret = %d)\r\n", ret);

bool errors = false;
while (d_stream.total_out < uncomprLen && d_stream.total_in < comprLen)
{
//d_stream.avail_in = d_stream.avail_out = 1; /* force small buffers */
ret = inflate(&d_stream, Z_NO_FLUSH);
if (ret == Z_STREAM_END) break;

if (ret != Z_OK)
{
printf ("ERROR: Unable to decompress deflate stream, error %d\r\n", ret);
errors = true;
break;
}
}

Out of curiosity, the P2 Demo space.iff begins like this:

00000000: 46 4F 52 4D 00 18 1F 5C 42 4F 4F 54 46 4F 52 4D FORM...\BOOTFORM
00000010: 00 17 EA 46 53 54 44 5F 53 48 50 31 00 04 E9 21 ...FSTD_SHP1...!
00000020: 44 65 66 21 32 11 0A 00 15 E9 04 00 78 DA EC DB Def!2.......x...
00000030: 87 5B 54 57 1A 06 F0 E9 85 DE 3B 0C 0C BD 77 A4 .[TW......;...w.
(...)

and it decompresses into:

00000000: 31 2E 34 30 2B 02 00 00 60 11 00 00 00 00 00 00 1.40+...`.......
00000010: 60 14 00 00 00 00 00 00 60 15 00 00 00 00 00 00 `.......`.......
00000020: 60 16 00 00 00 00 00 00 60 17 00 00 00 00 00 00 `.......`.......
00000030: 60 18 00 00 00 00 00 00 60 19 00 00 00 00 00 00 `.......`.......
00000040: 60 1A 00 00 00 00 00 00 60 1B 00 00 00 00 00 00 `.......`.......
00000050: 60 1C 00 00 00 00 00 00 60 1D 00 00 00 00 00 00 `.......`.......
00000060: 60 1E 00 00 00 00 00 00 60 1F 00 00 00 00 00 00 `.......`.......
00000070: 60 20 00 00 00 00 00 00 60 21 00 00 00 00 00 00 ` ......`!......
(...)

Which is good, there is definitely some structure there! At first glance it makes sense.

Without checking further, i suspect this particular chunk is a header that points to what may be sprite data. So you may want to parse the header and start dumping 8-bit bmps, to see if anything interesting comes up.

Anyway, hope this is useful! If you need the small VS project i hacked together for this test, let me know.

I'll see if I can find something on my old notes regarding the other algorithms...

 

[Lin Kuei Ominae]

This is great news HCl. Thank you.

Anyway, hope this is useful! If you need the small VS project i hacked together for this test, let me know.

Yep, this would be great. I could then directly convert it into C#, add it to the priv2editor and enhance it this way with a file-editing section.

Oh, this is great. I can't wait to finally have access to the raw data of priv2 and maybe be even able to change stuff.