TA3D hpi handlers.

Everything related to the code /
Tout ce qui touche au code
Post Reply
User avatar
Cire
Moderator
Posts: 350
Joined: Tue Oct 31, 2006 5:59 pm
Location: Somewhere on Earth

TA3D hpi handlers.

Post by Cire » Thu Nov 16, 2006 2:03 am

So I started to look at the hpi system used to depack files and what not, and I don't think the load sequence or priorty is correctly set. As well the code seems....bloated, and cumbersom, not to mention hard to follow for what it does.

From asking around the load sequence seems to be in priority, external files such as gaf in some directory persay, lets use maybe a data dir or something, that can look for externals, followed by gp3, ccx, hpi and ufo, for now I suggest we skip externals and just work with packed data. Priority should be gp3, ccx, hpi and then ufo. Thanks to Tro for the information.

I'am still debaiting on the best way to build a good hashed directory structure, however I believe that we should setup the play intro sequence to run seperatly in another thread while we do the work of building a decent directory structure (not loading files), as well as sorting the archive into a decent routine for quick lookup.


I'am also wondering if we shouln't add support for HPI v2, ala for future TAK support, its not much of a difference really, in fact its only an extra struct and looking at a version. This would also allow users to repack thir hpis using the better hpv2 format for TA3D, its just a thought.

++Cire.

User avatar
Cire
Moderator
Posts: 350
Joined: Tue Oct 31, 2006 5:59 pm
Location: Somewhere on Earth

Post by Cire » Fri Nov 17, 2006 5:32 am

I spent the last few days studing the hpi handling routines in the code, and researching somewhat.

I've written a class that should do everything we need within hpi, its much simpiler to use, and exposes only the methods that are needed to get the job done, so bloating should be miniable.

I can't take full creadit for the code however, as I borrowed quite a bit from the hpiview code released by Joe D, aka Kingbot I believe his name is, which is released to public domain.

Anyhow It now reads in all the hpis, similar to how it worked before, only more efficently written, and then processes directories and builds a list of files within the directories, overridding any files that should be used when appropiate, thus only decompressing whats needed unlike now. I havn't completely finished as I want to write file/directory to work similar to a fast hash lookup system for ultimate preforamce system.

This Most of this work (besides decompressing) should be able to be done while the application is displaying the intro, in fact it should only take 1/4 of the time needed, perhaps we could also decompess at that time a few key files that might be needed for menus, ect...i duno just guessing.

Anyhow i'd paste code onw but this editor would screw it up so i'll submit it directly to zuff and make code aviable from now on via urls from my website.

++Cire.

User avatar
Balthazar
Moderator
Posts: 2055
Joined: Wed Nov 01, 2006 4:31 pm
Location: Russian Federation
Contact:

Post by Balthazar » Fri Nov 17, 2006 6:34 am

Thanks, it sounds very nice :)

User avatar
zuzuf
Administrateur - Site Admin
Posts: 3281
Joined: Mon Oct 30, 2006 8:49 pm
Location: Toulouse, France
Contact:

Post by zuzuf » Fri Nov 17, 2006 10:34 pm

Yes that's a good idea, but preprocessing the directory structure doesn't seem necessary since it's done the first time you can load_file(...) and it's quite fast. But I agree that performance was not the main goal of my HPI module (the first thing I made in the project). HPI v2 support shouldn't be too hard to implement since it's an evolution of the first version.

I am having issues due to this module. I can't have the build option of a few units because there is a problem of priority of various files (.gp3, .hpi, .ufo, ...)

You can now post your code here, I deactivated HTML, it detected <asafsdfqs.h> as HTML code :x
=>;-D Penguin Powered

User avatar
Cire
Moderator
Posts: 350
Joined: Tue Oct 31, 2006 5:59 pm
Location: Somewhere on Earth

Post by Cire » Fri Nov 17, 2006 11:11 pm

OK I"ll try a submit. But please don't use them yet, i have still code to write out for cleaning up, ass well everything needs typecasted into proper use of variable sizes from our platform definations. Also, I have some windows specfic code atm that needs to be adjusted for allgero searchs and what not that you used before.

Anyhow heres the files, as I said I hope to have them finished tonight or tommorwo sometime, also I need to add more comments and provide a note of thanks to Joe D, the actual creator of the code to which I based alot of this on. Also note that I need to remove alot of crap that isn't needed anymore.

First: hpi.h

Code: Select all

#pragma once

namespace TA3D
{
	namespace UTILS
	{
		namespace HPI
		{
			#define HEX_HAPI 0x49504148
			#define HPI_V1 0x00010000

			#ifndef MAX_PATH
				#define MAX_PATH 260
			#endif
			class cHPIHandler
			{
			// Member Declarations:
			private:
				#pragma pack(1) // Byte alignment.

				struct HPIVERSION 
				{
					long HPIMarker; /* Must be HEX_HAPI */
					long Version;   /* Must be HPI_V1 */
				};

				struct HPIHEADER 
				{
					long DirectorySize; /* Directory size */
					long Key;           /* Decode key */
					long Start;         /* Directory offset */
				};

				struct HPIENTRY 
				{
					int NameOffset;
					int CountOffset;
					char Flag;
				};

				struct HPICHUNK 
				{
					long Marker;           /* always 0x48535153 (SQSH) */
					char Unknown1;         /* I have no idea what these mean */
					char CompMethod;       /* 1 = lz77, 2 = zlib */
					char Encrypt;          /* Is the chunk encrypted? */
					long CompressedSize;   /* the length of the compressed data */
					long DecompressedSize; /* the length of the decompressed data */
					long Checksum;         /* check sum */
				};

				struct HPIFILEDATA 
				{
					HPIHEADER H1;
					int Key;
					std::string HPIFileName;
					char *Directory;
					FILE *HPIFile;
				};

				struct HPIITEM 
				{
				  HPIFILEDATA *hfd;
				  HPIENTRY *E1;
				  int IsDir;
				  char *Name;
				  int Size;
				};
				#pragma pack()

			// Member Functions:
			private:
				void AddArchive( const std::string &FileName );
				void LocateAndReadFiles( const std::string &Path, const std::string &FileSearch );
				void ProcessRoot(HPIFILEDATA *hfd, const std::string &StartPath, int offset );
				void ProcessSubDir( HPIITEM *hi );

				void *GetMem( int size, int zero );
				int  ReadAndDecrypt(int fpos, unsigned char *buff, int buffsize);
				int ZLibDecompress(unsigned char *out, unsigned char *in, HPICHUNK *Chunk);
				int LZ77Decompress( unsigned char *out, unsigned char *in, HPICHUNK *Chunk);
				int Decompress(unsigned char *out, unsigned char *in, HPICHUNK *Chunk);
				void CloseTheFile(HPIFILEDATA *hfd);
				void cHPIHandler::CloseCurrentFile(void);


			public:
				void SearchDirForArchives( const std::string &Path );
				cHPIHandler::cHPIHandler() { HPIInfo=NULL; }
				cHPIHandler::cHPIHandler( const std::string &Path  ) 
				{ 
					HPIInfo=NULL;
					SearchDirForArchives( Path );
				}
				void ShowArchive();
				bool PullFromHPI( const std::string &FileName );

				// Member Variables:
			private:
				HPIFILEDATA *HPIInfo;
				std::string cDir;
				typedef std::map< std::string, HPIITEM *> HPIARCHIVE;
				HPIARCHIVE m_Archive;

			}; // class cHPIHandler;
		} // namespace HPI
	} // namespace utils
} // namespace TA3D
And the cpp

Code: Select all

//
//  HPI File Viewer
//

#pragma warning(disable: 4786)
#include <windows.h>
//#include <stdarg.h>
//#include <stdlib.h>
//#include <stdio.h>
#include <string>
#include <map>

#include <sys/utime.h>
#include "zlib.h"

using namespace std;

#include "hpiview.h"

#pragma comment(lib, "zlib.lib")


using namespace TA3D;
using namespace TA3D::UTILS::HPI;


void * cHPIHandler::GetMem(int size, int zero)
{
  void *result;

  if( zero )
     result = calloc( size, 1 );
  else
     result = malloc( size );

/*
  if( !result )
	   GlobalDebugger->Failed to alloc memory for hpi
*/
  return result; 
}

int  cHPIHandler::ReadAndDecrypt(int fpos, unsigned char *buff, int buffsize)
{
	int count, tkey, result;
	
	fseek(HPIInfo->HPIFile, fpos, SEEK_SET);

	result = fread(buff, buffsize, 1, HPIInfo->HPIFile);

	if (HPIInfo->Key)
	{
		for (count = 0; count < buffsize; count++) 
		{
			tkey = (fpos + count) ^ HPIInfo->Key;
			buff[count] = tkey ^ ~buff[count];
		}
	}

	return result;
}

int cHPIHandler::ZLibDecompress(unsigned char *out, unsigned char *in, HPICHUNK *Chunk)
{
	z_stream zs;
	int result;
	
	zs.next_in = in;
	zs.avail_in = Chunk->CompressedSize;
	zs.total_in = 0;
	
	zs.next_out = out;
	zs.avail_out = Chunk->DecompressedSize;
	zs.total_out = 0;
	
	zs.msg = NULL;
	zs.state = NULL;
	zs.zalloc = Z_NULL;
	zs.zfree = Z_NULL;
	zs.opaque = NULL;
	
	zs.data_type = Z_BINARY;
	zs.adler = 0;
	zs.reserved = 0;
	
	result = inflateInit(&zs);
	if (result != Z_OK) {
		//printf("Error on inflateInit %d\nMessage: %s\n", result, zs.msg);
		return 0;
	}
	
	result = inflate(&zs, Z_FINISH);
	if (result != Z_STREAM_END) {
		//printf("Error on inflate %d\nMessage: %s\n", result, zs.msg);
		zs.total_out = 0;
	}
	
	result = inflateEnd(&zs);
	if (result != Z_OK) {
		//printf("Error on inflateEnd %d\nMessage: %s\n", result, zs.msg);
		return 0;
	}
	
	return zs.total_out;
}

int cHPIHandler::LZ77Decompress( unsigned char *out, unsigned char *in, HPICHUNK *Chunk)
{
	int x, work1, work2, work3, inptr, outptr, count, done, DPtr;
	char DBuff[4096];
	
	done = false;
	
	inptr = 0;
	outptr = 0;
	work1 = 1;
	work2 = 1;
	work3 = in[inptr++];
	
	while (!done) 
	{
		if ((work2 & work3) == 0) 
		{
			out[outptr++] = in[inptr];
			DBuff[work1] = in[inptr];
			work1 = (work1 + 1) & 0xFFF;
			inptr++;
		}
		else 
		{
			count = *((unsigned short *) (in+inptr));
			inptr += 2;
			DPtr = count >> 4;
			if (DPtr == 0) 
				return outptr;
			else 
			{
				count = (count & 0x0f) + 2;
				if (count >= 0) 
				{
					for (x = 0; x < count; x++) 
					{
						out[outptr++] = DBuff[DPtr];
						DBuff[work1] = DBuff[DPtr];
						DPtr = (DPtr + 1) & 0xFFF;
						work1 = (work1 + 1) & 0xFFF;
					}
					
				}
			}
		}
		work2 *= 2;
		if (work2 & 0x0100) 
		{
			work2 = 1;
			work3 = in[inptr++];
		}
	}
	
	return outptr;
}

int cHPIHandler::Decompress(unsigned char *out, unsigned char *in, HPICHUNK *Chunk)
{
	int x, Checksum;
	
	Checksum = 0;
	for (x = 0; x < Chunk->CompressedSize; x++) {
		Checksum += (unsigned char) in[x];
		if (Chunk->Encrypt)
			in[x] = (in[x] - x) ^ x;
	}
	
	if (Chunk->Checksum != Checksum) {
		//		MsgPrintf(hwndMain, "Checksum error! Calculated: 0x%X  Actual: 0x%X\n", Checksum, Chunk->Checksum);
		return 0;
	}
	
	switch (Chunk->CompMethod) {
	case 1 : return LZ77Decompress(out, in, Chunk);
	case 2 : return ZLibDecompress(out, in, Chunk);
	default : return 0;
	}
}
/*
unsigned char *DecodeFileToMem(HPIITEM *hi)
{
  HPICHUNK *Chunk;
	long *DeSize;
	int DeCount;
	int DeLen;
	int x;
	unsigned char *DeBuff;
	unsigned char *WriteBuff;
	int WriteSize;
	int WritePtr;
  HPIENTRY *Entry;

	int Offset;
	int Length;
	char FileFlag;

	if (!hi)
		return NULL;

  Entry = hi->E1;

	Offset = *((int *) (HPIInfo->Directory + Entry->CountOffset));
	Length = *((int *) (HPIInfo->Directory + Entry->CountOffset + 4));
	FileFlag = *(HPIInfo->Directory + Entry->CountOffset + 8);

	WriteBuff = (unsigned char *)GetMem(Length+1, 0);
	if (!WriteBuff) {
//		MessageBox(hwndMain, "Unable to allocate extract buffer", szAppTitle, MB_OK);
		return NULL;
	}
	WriteBuff[Length] = 0;

	if (FileFlag) {
  	DeCount = Length / 65536;
  	if (Length % 65536)
		  DeCount++;
	  DeLen = DeCount * sizeof(int);

	  DeSize = (long *)GetMem(DeLen, 0);

  	ReadAndDecrypt(Offset, (unsigned char *) DeSize, DeLen);

	  Offset += DeLen;
	
	  WritePtr = 0;

	  for (x = 0; x < DeCount; x++) {
		  Chunk = (HPICHUNK *)GetMem(DeSize[x], 0);
		  ReadAndDecrypt(Offset, (unsigned char *) Chunk, DeSize[x]);
		  Offset += DeSize[x];

		  DeBuff = (unsigned char *) (Chunk+1);

		  WriteSize = Decompress(WriteBuff+WritePtr, DeBuff, Chunk);
		  WritePtr += WriteSize;

		  free(Chunk);
		}
	  free(DeSize);
	}
	else {
		// file not compressed
  	ReadAndDecrypt(Offset, WriteBuff, Length);
	}

	return WriteBuff;
}
*/

void cHPIHandler::CloseTheFile(HPIFILEDATA *hfd)
{
	if (!hfd)
		return;
	
	if (hfd->Directory) 
	{
		free(hfd->Directory);
		hfd->Directory = NULL;
	}


	if (hfd->HPIFile)
		fclose(hfd->HPIFile);

	hfd->HPIFile = NULL;
}

void cHPIHandler::CloseCurrentFile(void)
{
	if (HPIInfo)
		CloseTheFile(HPIInfo);
}

void cHPIHandler::ProcessSubDir( HPIITEM *hi )
{
	int *FileCount, *FileLength, *EntryOffset, *Entries;
	char *Name, *FileFlag, *ext;
	HPIENTRY *Base, *Entry;
	int count;
	char buff[256];
    HPIITEM *li;
	
	Base = hi->E1;
	if (Base)
		Entries = (int *) (HPIInfo->Directory + Base->CountOffset);
	else
		Entries = (int *) (HPIInfo->Directory + hi->hfd->H1.Start);
	
	EntryOffset = Entries + 1;
	Entry = (HPIENTRY *) (HPIInfo->Directory + *EntryOffset);
	
	for (count = 0; count < *Entries; count++) {
		Name = HPIInfo->Directory + Entry->NameOffset;
		FileCount = (int *) (HPIInfo->Directory + Entry->CountOffset);
		FileLength = FileCount + 1;
		
		li = (HPIITEM *)GetMem(sizeof(HPIITEM), 1);
		li->hfd = hi->hfd;
		li->Name = Name;
		li->E1 = Entry;
		
		if (Entry->Flag == 1) 
		{
			li->Size = 0;
			li->IsDir = 1;
		}
		else 
		{
			li->Size = *FileLength;
			li->IsDir = 0;
		}
		
		if (Entry->Flag == 1)	
		{
			std::string sDir = cDir;
		//	printf( "(DIR)%s\n", Name );
			cDir += Name;
			cDir += "\\";
			ProcessSubDir( li );
			cDir = sDir;

		}
		else {
			ext = strrchr(Name, '.');

			itoa(*FileLength, buff, 10);
			FileFlag = (char *)	(FileLength + 1);
			std::string f = cDir + Name;
			m_Archive[f] = li;
		//	printf( "%s%s LEN(%d)\n", cDir.c_str(), Name, (*FileLength) );
		}	
		Entry++;
	}
}

void  cHPIHandler::ProcessRoot(HPIFILEDATA *hfd, const std::string &StartPath, int offset )
{
	int *Entries, *FileCount, *EntryOffset;
	char *Name;
	std::string MyPath;

	HPIInfo = hfd;

	Entries = (int *)(hfd->Directory + offset);
	EntryOffset = Entries + 1;
	HPIENTRY *Entry = (HPIENTRY *)(hfd->Directory + *EntryOffset);

	for( int count = 0; count < *Entries; count++ ) 
	{
		Name = hfd->Directory + Entry->NameOffset;
		FileCount = (int *) (hfd->Directory + Entry->CountOffset);
		if (Entry->Flag == 1)	
		{
			MyPath = StartPath;
			if( MyPath.length() )
				MyPath += "\\";
			MyPath += Name;
			cDir = MyPath + "\\";

		//	printf( "%s\n", MyPath.c_str() );

			HPIITEM *hi = (HPIITEM *)GetMem(sizeof(HPIITEM), 1);
				
			hi->hfd = hfd;
			hi->IsDir = 1;
			hi->Size = 0;
			hi->Name = Name;
			hi->E1 = Entry;
				
			ProcessSubDir( hi );
		}
		Entry++;
	}
}



// start new code.
void cHPIHandler::AddArchive( const std::string &FileName )
{

	HPIFILEDATA *hfd = (HPIFILEDATA *)GetMem(sizeof(HPIFILEDATA), 1);
	if (!hfd) 
		return; // No Need to generate error here as getmem already did it.

	hfd->HPIFileName = FileName;
	
	if( !(hfd->HPIFile = fopen(hfd->HPIFileName.c_str(), "rb")) )
	{
		CloseTheFile(hfd);
		free(hfd);
//		GlobalDebugger-> Failed to open hpi file for reading.
		return;
	}

	HPIVERSION hv;
	fread(&hv, sizeof(HPIVERSION), 1, hfd->HPIFile);
	
	if( hv.Version != HPI_V1 || hv.HPIMarker != HEX_HAPI )
	{
		// No need to generate error I don't think, just wasn't an
		//   hpi archive we were interested in, possibly packed with
		//   version 2, or was a save file.
		CloseTheFile(hfd);
		free(hfd);
		
		hfd = NULL;
	}


	fread(&hfd->H1, sizeof(HPIHEADER), 1, hfd->HPIFile);
	if (hfd->H1.Key)
		hfd->Key = ~((hfd->H1.Key * 4)	| (hfd->H1.Key >> 6));
	else
		hfd->Key = 0;

	int start = hfd->H1.Start;
	int size = hfd->H1.DirectorySize;
	
	hfd->Directory = (char *)GetMem(size, 1);
	HPIInfo = hfd;
		
	ReadAndDecrypt(start, (unsigned char *)hfd->Directory + start, size - start);
	cDir = "";

	ProcessRoot(hfd, "", start);
}

void cHPIHandler::LocateAndReadFiles( const std::string &Path, const std::string &FileSearch )
{
	WIN32_FIND_DATA FindFileData;
	HANDLE hFind = INVALID_HANDLE_VALUE;

	if( (hFind = FindFirstFile( FileSearch.c_str(), &FindFileData))==INVALID_HANDLE_VALUE) 
      return;
  
	AddArchive( Path + FindFileData.cFileName );
      
   while (FindNextFile(hFind, &FindFileData) != 0)
	   	AddArchive( Path + FindFileData.cFileName );
    
   DWORD dwError = GetLastError();
   FindClose(hFind);

//   if (dwError != ERROR_NO_MORE_FILES) 
//      GlobalDebugger->"While searching for files got error, dwError;
}

void cHPIHandler::SearchDirForArchives( const std::string &Path )
{
	const char ext[4][6] = { "*.ufo", "*.hpi", "*.ccx", "*.gp3" };

	for( int i = 0; i < 4; i++ )
		LocateAndReadFiles( Path, Path + (char *)ext[i] );
}

void cHPIHandler::ShowArchive()
{

	for(HPIARCHIVE::iterator iter = m_Archive.begin(); iter != m_Archive.end(); iter++)
	{
		printf( "(arch)%s\n", (*iter).first.c_str() );

	}	
}

// if success cast to uchar
bool cHPIHandler::PullFromHPI( const std::string &FileName )
{

	HPIARCHIVE::const_iterator iterFind = m_Archive.find( FileName );
	if( iterFind != m_Archive.end() )
	{
		HPIITEM *hi = iterFind->second;
// todo: check of hpi chunk is null if so decode
//         otherwize return chunk buff, this function should also return
//         void *, if null no fileexisted, otherwize cast the value to
//         uchar (unsigned char *)
		return true;

	}
	return false;
}

// main, only used for testing.
int main(int argc, char* argv[])
{
	cHPIHandler cHPI( "C:\\Cavedog\\Totala\\" );

	bool test;
	test = cHPI.PullFromHPI( "units\\ARMFARK.FBI" );

//	CloseAllFiles();


	return 0;
}



Post Reply

Who is online

Users browsing this forum: No registered users and 34 guests