Topic: Partial (yet extensive) rewrite of the PlayStation port

[Gemini]

Time for a comeback! So here I go again with my hacking of FF4 for the PSX, this time with some interesting new approach to the task, something that could also be a group effort because of the way I'm managing it at the moment. Briefly, I'm working on a code rewrite of the engine built on the PSX port made by TOSE, with the reworked code written almost entirely in C.

What does this mean? More power and ease of writing the code. With a C framework the hacking of this game is quite faster to carry on, as you can see from these pictures:

These are to show you my personal reimplementation of the menu shell, completely made from scratch. Menus implemented so far are the following:

• Root;
• Inventory;
• Abilities/Spell/Magic (this one has a little gimmick enabling the user to pick any spells from the list, even monster commands);
• Status.

Visually speaking they are completely functional, however they lack most of user interaction. I'll be working on this as soon as I'm done gathering more information about how the game manages some of the key fields in data structures (getting that "To Next Level" value was a tough one; kudos to pinkpuff for the help!). Square did a real mess with data storage in this game. :/ I'll probably expand and rearrange some of the data if necessary.

What has been done so far on the coding side of the project:

• Updated graphics, a combination of DS and GBA assets;
• Font sets are rendered proportionally and with arbitrary screen coordinates, allowing much more text to fit and to be placed anywhere you wish;
• Most text assets have been expanded and no longer pose any real limitations, removing entirely the issues caused by fixed length strings. They can also be arranged in packages, allowing support for multiple languages like on the GBA port (you can see this from the mixed languages in my screens);
• Playable character names have been expanded to 8 letters with a simple 6-bit encoding;
• Dialog banks are compressed using MultiHuffman, which means no byte goes to waste. ;P

And here's a couple older hacks I made in full assembly (I'm already rewriting most of these to C, don't worry):

I wouldn't really dislike replacing the entire GFX with that of the GBA port, which would not just improve the generally poor quality but also fix bugs with most emulators (notice the black stuff pretty much everywhere a transparency should apply). Also replacing the streaming sound engine with a proper tracker wouldn't be too bad, but I still need to decide whether to use the DS soundtrack (to which I already have a partially complete player) or the original one, maybe with slightly updated instruments. This last change would also allow for data to freely come from the CD-Rom unit, instead of using a huge block of data as a straight copy from the SNES rom.

My project would also include a full rewrite of some of the trivial stuff posed by hacking this game, such as spell formulas. This should give a hacker enough room to hack the engine in an easier way so that custom parts could be added at will with very little effort. For example, you'd be able to add extra spells, enemies, new commands, and change the general behaviour of the engine itself.

Everyone's free to contribute with code or information to build a more complete framework! :wink: Yet, the former contribution form isn't ready to go as I need to properly put up a google code page. :P An old revision of the code can already be found here, but I'll provide soon a link with the latest stuff.

[Grimoire LD]

This is an an absolutely, positively, incredible undertaking you are on Gemini! so this would be an executable of FFIV then with all of the code written in C for people (with knowledge of C) to pick up, alter and add as they please then?

I must say though... I absolutely Hate the GBA Portraits, they looked Way too Western, (FFIV DS only a slight improvement in that regard) I personally think that FFIV for the SNES or FFIV Complete Collection had the best portraits.

Other than that I love how slick and smooth everything looks in the menus!

[LordGarmonde]

Hi Gemini! I remember seeing your early posts with pics of your first editor; impressive! 

Time for a comeback! So here I go again with my hacking of FF4 for the PSX, this time with some interesting new approach to the task, something that could also be a group effort because of the way I'm managing it at the moment. Briefly, I'm working on a code rewrite of the engine built on the PSX port made by TOSE, with the reworked code written almost entirely in C.

What does this mean? More power and ease of writing the code. With a C framework the hacking of this game is quite faster to carry on...

To quote Eric Cartman: "You bet your sweet clown ass it is!" - now C is something I have some history with

I'm in the midst of attacking the experience problem myself - doing away with the reference table all together and calculating it. But with C I can easily write up a formula to calculate TNL from current level. You can see what's been cooking on that regard if you are interested - but the C version would be infinitely simpler:
      TNL (for Lv. X) = aX⁴ + bX³ + cX² +dX + e  -- or something like that: very little to ask of C! 

Anyway if you're interested I'll write it up sooner rather than later (I know I'll write it up eventually just 'cus) - It's good to see you here; to you I wish the very best of luck!

Oh to be able to ask of Assembly what I can C - but, challenge accepted!

http://slickproductions.org/forum/index.php?topic=1905.msg20089#msg20089

[Pinkpuff]

First off, this looks totally amazing.

Sadly though, using FF4kster with this will almost certainly be disastrous for your rom.

And... oh God, Huffman trees... I definitely understand the utility of them but I still get nightmares about those from trying to hack Jeopardy for the NES. I managed to figure out the tree by tracing the code, then found that the answers used a different tree from the clues... and then once I deciphered those I discovered I had no clue how to even go about looking for the pointers... it was a mess. But at least with this, you'll have that kind of information handy right? 

[Gemini]

This is an an absolutely, positively, incredible undertaking you are on Gemini! so this would be an executable of FFIV then with all of the code written in C for people (with knowledge of C) to pick up, alter and add as they please then?

It creates a bunch of compiled binaries that can interact with the rest of the game thanks to armips' integration in assembly for hooking up the new code. It will still use the original executable for the most and original data when needed, but all the extra code will reside in separate and compact modules, and that's where the user can interact to make huge modifications to the game's behaviour.

I must say though... I absolutely Hate the GBA Portraits, they looked Way too Western, (FFIV DS only a slight improvement in that regard) I personally think that FFIV for the SNES or FFIV Complete Collection had the best portraits.

Portraits can be easily replaced with anything really. At the moment I'm simply converting them on-the-fly from a file stored with the SNES 4BPP encoding, but this can be changed to use proper bitmap images, even 8 bit ones if necessary. Branching from here is extremely simple because I made so that tiled pictures are assembled in a non-tiled manner and replacing sprite information is just a matter of populating a mapping structure. :D

I'm in the midst of attacking the experience problem myself - doing away with the reference table all together and calculating it. But with C I can easily write up a formula to calculate TNL from current level. You can see what's been cooking on that regard if you are interested - but the C version would be infinitely simpler:
      TNL (for Lv. X) = aX⁴ + bX³ + cX² +dX + e  -- or something like that: very little to ask of C! 

Calculating said value in C was pretty much straightforward, other than for a few quirks with the way FF4 manages base level for a character. :P It took me a while to figure out what Square did there with those pointers. As for the TNL formula, this is what it looks like in my code:

Code: [Select]

// resolve the huge mess needed for calculating "to next level" values
int Get_exp_to_next(PLAYER_DATA *p)
{
u32 cur_exp, prev_exp, need_exp;
int cur_lv;
LEVEL_UP_DATA *lv;
u16 *ptr=(u16*)levelup_data;

// take into account levels >= 70
cur_lv=(p->level>=70 ? 69 : p->level-1);
// pointer to this char's level up table
// NOTE: pointers are coded in a way that makes the base level useless because
// 'garbage' level up data is counter already as skipped, meaning the pointer
// actually counts from a point way below the start level
lv=(LEVEL_UP_DATA*)&levelup_data[ptr[p->id-1]+cur_lv*sizeof(LEVEL_UP_DATA)-0xB500];
// cache variables
cur_exp=MAKE24WORD(p->exp);
prev_exp=MAKE24WORD(p->exp_prev);
need_exp=MAKEHWORD(lv->exp) | ((lv->mp_gain & 0xE0)<<11);

// retrieve the correct value
return prev_exp+need_exp-cur_exp;
}Quite small, isn't it? And since there is a ton of room for new data, this could be expanded to make full level up tables for the entire cast, not just "start from lv X" ones.

Sadly though, using FF4kster with this will almost certainly be disastrous for your rom.

Yeah, I thought of this too, but there is a solution, although it could be awkward to realize: databases. Say, you could save most information in an expandable format as human readable data, like XML. You could dump data to XML and graphics to PNG/BMP, then open a project from there and transform the data into whatever form you desire, being it for the original SNES rom or smaller packages needed by my project. I'll give this a shot soon in order to dump all monster data to a database I will later use for a bestiary; I'll be using a structure that embeds all information for a monster in a single block of data with no shared attributes (i.e. drop lists can be updated to be unique).

And... oh God, Huffman trees... I definitely understand the utility of them but I still get nightmares about those from trying to hack Jeopardy for the NES. I managed to figure out the tree by tracing the code, then found that the answers used a different tree from the clues... and then once I deciphered those I discovered I had no clue how to even go about looking for the pointers... it was a mess. But at least with this, you'll have that kind of information handy right? 

Mind you, I don't have any idea how this works exactly as I'm only familiar with regular Huffman. xD Kingcom sent me a wall of C/C++ code a long time ago with his own implementation, to test how MultiHuffman would perform on FF4; it did pretty well, even with no DTE or dictionary, but it's only natural as this algorithm was specifically intended to be used on text rather than any other types of data. Surprisingly, the actual decompression code used in game is quite small, like 50-55 lines of assembly at best.

[Pinkpuff]

Sadly though, using FF4kster with this will almost certainly be disastrous for your rom.

Yeah, I thought of this too, but there is a solution, although it could be awkward to realize: databases. Say, you could save most information in an expandable format as human readable data, like XML. You could dump data to XML and graphics to PNG/BMP, then open a project from there and transform the data into whatever form you desire, being it for the original SNES rom or smaller packages needed by my project. I'll give this a shot soon in order to dump all monster data to a database I will later use for a bestiary; I'll be using a structure that embeds all information for a monster in a single block of data with no shared attributes (i.e. drop lists can be updated to be unique).

Aw yeah, I love it! Just to make sure I understand what you're saying correctly, we could have some kind of agreed-upon "save file" format, different/separate from the actual rom, that both our programs could import/export from?

And... oh God, Huffman trees... I definitely understand the utility of them but I still get nightmares about those from trying to hack Jeopardy for the NES. I managed to figure out the tree by tracing the code, then found that the answers used a different tree from the clues... and then once I deciphered those I discovered I had no clue how to even go about looking for the pointers... it was a mess. But at least with this, you'll have that kind of information handy right? 

Mind you, I don't have any idea how this works exactly as I'm only familiar with regular Huffman. xD Kingcom sent me a wall of C/C++ code a long time ago with his own implementation, to test how MultiHuffman would perform on FF4; it did pretty well, even with no DTE or dictionary, but it's only natural as this algorithm was specifically intended to be used on text rather than any other types of data. Surprisingly, the actual decompression code used in game is quite small, like 50-55 lines of assembly at best.

Oh yeah don't get me wrong, Huffman is super efficient! I know nothing about "MultiHuffman" but I assume it's probably even more efficient. But it's just such a pain for hacking ^_^;

[LordGarmonde]

Quite small, isn't it? And since there is a ton of room for new data, this could be expanded to make full level up tables for the entire cast, not just "start from lv X" ones.

Very true - that's nice bit of code - and with how simply that call is and space at your disposal I think you're right not to screw around. I'll confess that even in the original ROM with space constraints as they are - had it been a complete table I probably would have just let it go. On the plus side, since Square went all Kung-Fu Cut Man on it ('yikes' on that reference...) I had already filled in the blanks. Values should be pretty accurate - I used the averages from each character that actually had data for that interval (i.e. Paladin Cecil and Rydia for 1-4, then those two and Edward up through 9, etc.) but each value was adjusted based on how the compared to each other over they later intervals where the data did exist. Enjoy!

[Gemini]

Aw yeah, I love it! Just to make sure I understand what you're saying correctly, we could have some kind of agreed-upon "save file" format, different/separate from the actual rom, that both our programs could import/export from?

Pretty much, yeah. With XML you define a more or less standard way to represent data, which can be manually edited and expanded at will. Here's and example of what I have in my enemy database:

Code: [Select]

<Monsters>
  <Monster ID="0" Name="Imp">
    <Stats lv="3" hp="6" boss="0" atk="1" def="96" mdef="160" spd="2"/>
    <Status type="0" atk_elem="0x00" atk_stat="0x0000" def_elem="0x00" def_stat="0x0000" weakness="0x00"/>
    <Gain exp="28" gil="5"/>
    <Drops drop_1="206" drop_2="226" drop_3="187" drop_4="231" drop_rate="1"/>
  </Monster>
  <Monster ID="1" Name="Basilisk">
    <Stats lv="5" hp="90" boss="0" atk="6" def="98" mdef="166" spd="6"/>
    <Status type="4" atk_elem="0x00" atk_stat="0x0000" def_elem="0x00" def_stat="0x0018" weakness="0x00"/>
    <Gain exp="110" gil="30"/>
    <Drops drop_1="213" drop_2="213" drop_3="213" drop_4="213" drop_rate="1"/>
  </Monster>
  <Monster ID="2" Name="Eagle">
    <Stats lv="5" hp="18" boss="0" atk="3" def="96" mdef="160" spd="2"/>
    <Status type="0" atk_elem="0x00" atk_stat="0x0000" def_elem="0x00" def_stat="0x0038" weakness="0x20"/>
    <Gain exp="40" gil="5"/>
    <Drops drop_1="212" drop_2="213" drop_3="213" drop_4="233" drop_rate="1"/>
  </Monster>
</Monsters>Forgive the horrible arrangement, still working on it.

Oh yeah don't get me wrong, Huffman is super efficient! I know nothing about "MultiHuffman" but I assume it's probably even more efficient. But it's just such a pain for hacking ^_^;

MultiHuffman is even more efficient as it uses a set of trees for each text bank, and each tree can branch to a different one depending on the contents of the string. In other words, Huffman on steroids.

Values should be pretty accurate - I used the averages from each character that actually had data for that interval (i.e. Paladin Cecil and Rydia for 1-4, then those two and Edward up through 9, etc.) but each value was adjusted based on how the compared to each other over they later intervals where the data did exist. Enjoy!

If those can be reformatted to a database of sorts I'm sure we can put them to good use.

[LordGarmonde]

If those can be reformatted to a database of sorts I'm sure we can put them to good use.

I'll need to work up an XDS map for Excel to use in exporting the table as XML file. In the meantime the link below leads to a post on The Drawing Board with the whole table if you want to quickly grab/reference  a number

http://slickproductions.org/forum/index.php?topic=1911.msg20136#msg20136

[Pinkpuff]

Pretty much, yeah. With XML you define a more or less standard way to represent data, which can be manually edited and expanded at will. Here's and example of what I have in my enemy database:

Yup that's pretty much what I was picturing. FF4kster will still need a ROM file regardless because it gets all sorts of other information from there, including things like the font and menu graphics (so it's literally unusable without a loaded rom). But I can certainly picture something like "export project file" which will create an xml file based on the loaded rom or "import project file" which will read the xml file the user provides and alter the loaded rom according to the specifications.

That said, it probably won't be happening any time soon. It's still evolving at a fairly rapid pace (relatively speaking), and we're still discovering things about the game data that would cause us to have to revisit the structure for some objects fairly frequently. But it does sound like an ideal solution to the problem down the road.

[Gemini]

Having fun with a bestiary module:

It's still a WIP, so don't mind the horrid layout and strings inconstantly flying all over the screen. I took some time to add a few things just for this one, that is archived files (instead of having a ton floating on my folders) used to store monster data for this. Currently this is using the GBA sprites for monsters and, again, PSP backgrounds (resized a little better than the older ones). Drop rates are recalculated so that repeated items stack in a single entry and generate a global percentage. Say, if you have 3 potions for a 25% drop rate, there will be only one potion showing in there with a drop rate of 18.75%.

Now, if I could get those threads to work this could use a lot of async stuff.

I'll need to work up an XDS map for Excel to use in exporting the table as XML file. In the meantime the link below leads to a post on The Drawing Board with the whole table if you want to quickly grab/reference  a number

I guess I'll take a look at it. Thanks!

But I can certainly picture something like "export project file" which will create an xml file based on the loaded rom or "import project file" which will read the xml file the user provides and alter the loaded rom according to the specifications.

That's the same thing I'm doing right now, in a manner of speaking. Currently dumping some of the data to XML, then I'll start adding tweaks. The bestiary thingy already contains some of these tweaks, like a description and custom fields.

That said, it probably won't be happening any time soon. It's still evolving at a fairly rapid pace (relatively speaking), and we're still discovering things about the game data that would cause us to have to revisit the structure for some objects fairly frequently. But it does sound like an ideal solution to the problem down the road.

No problem, there's no rush.

[JCE3000GT]

Definitely cool.  I will follow this for sure.  :)

[Dragonsbrethren]

Wow, this is incredible work, Gemini!

[Gemini]

I guess I'm getting kinda frustrated again by how this game stored every damn piece of information. On to the problem:

Now, while most of it is fine in my revision of the bestiary, there is something missing: the 'Magic' attribute. I have absolutely no idea where that is stored or how it could be calculated, since it's not part of the usual stats a monster has.

Here's what Rubicante's stats look like in my XML:

Code: [Select]

  <Monster ID="187" Name="Rubicant">
    <Stats lv="50" hp="34000" boss="1"/>
    <Attack mult="8" perc="70" base="80"/>
    <Defense mult="2" perc="25" base="3"/>
    <MagicDefense mult="8" perc="80" base="37"/>
    <Speed min="38" max="38"/>
    <Status type="0" atk_elem="0x00" atk_stat="0x0000" def_elem="0x41" def_stat="0x0400" weakness="0x00"/>
    <Gain exp="18000" gil="7000"/>
    <Drops id1="206" id2="206" id3="206" id4="206" rate="0"/>
  </Monster>
Could the magic value be derived from somewhere else? Like the attack patterns and stolen items. No clue really, but I'd like this to be as complete as possible, so if anybody can help let me know.

Meawhile I started adding more to the module itself, like the ability to switch backgrounds on-the-fly and a full scrollable list (a carbon copy of the usual bestiaries from FF Origins on). The information for the BGs is stored along with the order of the bestiary, meaning we can have the list to skip bogus entries such as scripted battles or dummy enemies.

[chillyfeez]

Do you mean magic as in, what spells he can cast (as opposed to his magic power, I mean)?
If so... hoo boy, are you in for a treat...