Topic: Chillyfeez's Shadow Party Hack V 2.0

[chillyfeez]

Hello, interested parties!

This is a thread where I'll be documenting the progress of a new "side project." Hopefully this will eventually become the definitive SNES FFIV "everyone gets stored in the Shadow Party" mod.

At this point, it's all conceptual and theoretical, but I'm pretty sure that, with enough time, I can make this mod work exactly according to the plan I'll describe here.

The goal of this project is to make the following changes to the Vanilla FFIIUS ROM:
1) All characters except Cecil can be stored in the shadow party data. As a general rule, characters in the shadow party will not earn experience.

2) Shadow party data, which will occupy the same amount of space in SRAM as it does in Vanilla, will be divided into two distinct parts. Part 1 (0x80 bytes) will store two full character records. Characters stored in part 1 will earn experience. By default, these full records will be used to store Kain (when he is under Golbez's control) and Rydia (between the shipwreck and her return as a teenager). Part 2 (0xB0 bytes) will store a set of essential data for Kain, Rydia, Tellah, Edward, Rosa, Yang, Palom, Porom, Cid, Edge and FySoYa. Characters stored in this section will not gain experience. The tradeoff is that all characters can leave and rejoin the party without losing stat information.

3) The Namingway screen will be removed and replaced with a party switch system, which I am calling the PHS (after the system in FFVII)

4) Character equipment will not be stored in shadow party data, but a hacker using this mod will be able to choose whether all equipment is lost forever or added to the player's inventory (or the fat chocobo's inventory?) when a character is moved to the shadow.


I think this mod will be interesting enough on its own that FFIV fans might want to try it just for fun. The great part about it, though, is that it will be fully compatible with FF4kster, so it will easily lend itself to others' hacking projects.

A few preemptive answers to some possible questions...
Why can't I store Cecil?
Essentially, I had to draw the line somewhere. The reason it's so hard to make a mod that fully stores every character in the shadow party is that the game only has 320 bytes available for the shadow party. I need 128 of those bytes for the records that can gain experience (I wanted 192, actually, but again I had to draw the line), which leaves me with only 192 for the non-experience slots. Even using a little compression trick I thought up, I still need 16 bytes for each character slot, and 16 x 11 (the total number of non-Cecil characters) = 176. I still have 16 bytes leftover then that could be used for Cecil, but that leaves open the possibility to face Zeromus without the ability to use the Crystal, which would leave the player in a game breaking dead end.

So then why include records that can gain experience at all?
In the default game story, I think it's necessary that Kain and Rydia gain experience when they are away. Story-wise, they are off actually doing things (as opposed to, say, Rosa, who spends her time away from the party completely tied up). And also, they are gone long enough that if they didn't level up, their return would be anticlimactic. Kain has, by far, the longest period of time away from the party. You don't want him rejoining at level 13. Rydia needs to come back and immediately provide significant help defeating Golbez and his shadow dragon. In a perfect world, Yang would level up, too, but there isn't enough room, and at least he's only gone for one chapter of the story so he doesn't miss out on very much levelling.

What are these "essential stats" that are getting stored?
A large portion of the character record is completely static and easily retrievable from ROM. What will be stored is everything that is not, specifically: Level, Persistent Negative Statuses, Current HP, Max HP, Current MP, Max MP, Base Str, Base Agi, Base Vit, Base Wis, Base Will, and Exp. I could have opted to save space by not including Persistent Statuses, Current HP and Current MP, but then the PHS system could be used as a free Cabin (for everyone except Cecil) and I feel like that would be a major flaw. The choice not to store equipment also saves seven bytes per record.

But wait - I just counted, and that all adds up to 18 bytes, not 16 - what's up with that?
You're absolutely right. I devised a bit of a compression trick. It's not much, but it saves two bytes per record. The highest possible value for Current and Max HP is 9999, or 0x270F, or #00100111 00001111. The highest possible value for Current and Max MP is 999, or 0x03E7, or #00000011 11100111. So HP never uses the two most significant bits, and MP never uses the six most significant bits. By shifting Max HP left by two bits I can combine it with Max MP into three instead of four bytes, and likewise for Current HP and MP.

So what happens with the stored characters' equipment?
As I said, the hacker will have the option when a character "leaves" to either discard current equipment outright, or to move the equipment into the player's inventory. To me, there are some instances when it makes sense in the story just to discard the equipment - specifically when a character leaves against the party's will (e.g. - Kain gets hypnotized, party gets shipwrecked); and there are times when a departing character might be able to leave equipment with the party (Palom & Porom get stoned, Cid stays in the Dwarf kingdom, any time PHS is used). I'm pretty sure I can work it out that if the player's inventory is full, the "swap or discard" screen will kick in, though I haven't actually tried yet...

So you're hacking the Namingway screen... How's that gonna work?
Well, I don't exactly know just yet. I can tell you that the player will lose the option to change Cecil's name to "Shitty," but the tradeoff should be worth it. As for how I actually come around to hacking the screen... Well you, dear reader, get to see that develop right before your very eyes. You see, I have a concept of how it will all work, but I don't know enough about the SNES PPU to code the visuals yet. Squall is helping me figure that all out on this forum, so you can watch as I learn what I need to learn.

Stay tuned!

[chillyfeez]

If you want to catch up on the discussion between Squall and myself thus far regarding hacking the Namingway screen, you can find it here.

[chillyfeez]

Please don't alter the $2101, just read the value written there, particularly bits 5-7 (sets Sprite size).
Post the code here, so we can delve in details. Keep in mind that reading/writing to video could be auto-incremented. So even if you have initial value in the address ($2116), if you have some read/writes ($2118/$2139) after initialization and the code you are looking, the address has been already changed! Besides as I said the transfer could be done trough DMA/HDMA ... it all depends.

You know what? Lets do something else:
 - Download vSNES (the tool I've talk about)
 - Make sure you use zSNES as emulator. Load the ROM and when the namingway screen pops up - make a savestate
 - Send me or attach here that savestate
* You can check for yourself many things in vSNES - like what mode you use, layers, tiles, sprites,...

Ok, I made a save state in zSNES and downloaded vSNES and opened it but I don't really know what I'm looking at here...
It's too large to attach so here's a link:
https://drive.google.com/file/d/1_HmFLAMLgSGUHpSmtYkM8ses9BuAsiqD/view?usp=drivesdk

[Squall]

First is the blue window. You can see pretty much all register include PPUs. From there we can learn the mode, the layers,... but you need a good 'help'. I will suggest this

Although we have the full info, and we can check the ASM code, its a little bit hard to get info, its more for a reference. Click on 'Screen Viwer', the last tab is 'info'. This is where the info from registers is made in a human form :D

1. Always the first things to check in is mode: here is 0. If you check the doc I linked above, you will see that in that mode we have 4 layers (Backgrounds/BG) all of them 2bpp or 4 colors. This mode is good for simple graphic or heavy on text. Also its important to note screen size: 256x224
2. Second is layers (backgrounds) - you will see that only BG1 and BG4 are visible, the other two - not. 'tile base' is where in the VRAM is the address of tiles, 'tilemap base' is the actual layer content.  I will suggest to click on 'screen' tab and play with layers visibility. Its crucial to understand what element in what layer goes and why!!! In 'layers' tab you can preview each leayer individually, examine what tiles compose the image (just move the mouse over preview).
3. Same way check sprites - in 'info' section about their properties, and in 'layers|sprites' - preview

*Some interesting conclusions:
- although mode 0 is quite limited of colors, trough sprite layer you can add some colors to the screen (since sprite layer is always 4bb or 16 colors). I think it will work fine for your PHS idea: - character that need to be moved will be Sprites, and you can use the screen to show some of their stats
- Its seems to me that whoever has made this, instead of defining a char as a Sprite he/she have defined each tile as Sprite ... I don't see the point of doing it that way ... logically you should have only 4 sprites (1 for the pointer, 2 for the chars and 1 for portrait)
- by knowing ID of tiles, you may try to check the code for LDA/x/y ID and pinpoint 'the drawing' on the screen

[chillyfeez]

- Its seems to me that whoever has made this, instead of defining a char as a Sprite he/she have defined each tile as Sprite ... I don't see the point of doing it that way ... logically you should have only 4 sprites (1 for the pointer, 2 for the chars and 1 for portrait)

Yeah, FFIV was the first game Square developed for SNES, and it was released only about eight months after the console. A lot of times when you dig into the code, you get the impression that they really didn't understand the capabilities or how best to use the hardware yet.

- by knowing ID of tiles, you may try to check the code for LDA/x/y ID and pinpoint 'the drawing' on the screen

Looking forward to that for the practical application (I mean, this project), but I also still want to understand better how this all really works. So, thank you for taking this time.

I have to digest everything here. It's a busy work weekend for me (do they do "Black Friday" where you are?), but I have Monday and Tuesday off, so we can pick up then at the latest.

[Squall]

Writing those long paragraphs take me quite sometime - to check, to experiment, to synthesize, to add some practicality ... so I really need your input - is it too much, is it not enough, whatever is your first impression/response after reading it (if you prefer PM, Telegram thats fine too)!

[chillyfeez]

It's been a good amount of info at a time. I like having it up on the forum, too, because then I know it will be there if I need to go back and reference it again sometime.

[chillyfeez]

OK, I had some time to do some studying this evening and I had a bit of a breakthrough...

So it seems that FFIV always stores object data in RAM at $0300-$0520 and then regularly does what I think is a DMA transfer. I don't really know what DMA is, but I'm pretty sure that's what's happening.

So that's all well and good. By playing with the data there, I can effectively move the Namingway image to the location where I would want a portrait to be, giving me proof of concept that I could put portraits on the screen in all the right spots.
Here's the problem I'm facing now, though: Maybe it's because the game is treating every tile like a sprite, instead of just using 32x32 sprites, but that Namingway image uses 0x40 bytes of data. So if I wanted to display 11 character portraits on screen at a time, I'd need to transfer 0x02C0 bytes instead of 0x0220. Add in the finger cursor, and I'd need 0x02D0. Then if I also wanted the current party's battle sprites displayed, I'd need 0x0370. I think I could condense all the space used down to 0x0330 bytes, but that doesn't matter, because it seems like the game is already using the RAM beginning at $0600 for something else, so unfortunately it looks like I'm going to have to do without the battle sprites.

The question, then, is this: is there any reason (hardware limitations, or anything like that) why I can't change the 220-byte DMA transfer to a 2D0-byte DMA transfer? I've found the line of assembly that sets the transfer at 220 bytes, so the change would be simple,as long as it wouldn't cause a crash.
The other thing I could learn how to do, I guess, would be to change it so that the game is using 32x32 sprites... Or at least 16x16 sprites. Based on what I'm reading, it seems like that might be the better option, because the Battle sprites are 16x24, so I'd need to keep them as collections of 8x8 sprites, but then each portrait could be four 16x16 sprites instead of 16 8x8 sprites.

 
OK, forget most of the above post. I just figured out that when they say OAM is 512 + 32 bytes, that's a decimal value, so those 0x0220 bytes being transferred are the entirety of the OAM. That means I have to figure out how to make those sprites 16x16 in order for this to work. By manipulating those last 0x20 bytes I can make the Namingway sprites load as 16x16, but they're not composed of the right tiles (they're half Namingway and half NPC merchant), so I have to figure out the proper arrangement of data in VRAM in order to make the sprites look right. I'm sure there's a pattern and it'll be pretty obvious once I delve into it...

[Squall]

II. Layers
The second abstraction is Layer. You may see it as Background or BG in documentation. It work the same way  as in specialized graphic applications: Photoshop or Gimp - the final image is composed by plotting couple of images 1 by 1, each on top of the next, each containing transparent pixels trough which you may see things from the image under it.

SNES contain 4 such layers  (usually called background) + 1 sprite layer (which function in similar way) for a total of 5 layers. Unlike Photoshop where each layer can have 256 levels of transparency, here we have only 2 levels - fully transparent or not. Using color with index 0 mark transparent parts.

The other difference from Photoshop is that in each layer we can define priority for individual Tiles. For BG1-4 we have priority 0 and 1, while for Sprites (OBJ) we have 0-3. The order in which final image is rendered depend on the mode, so check the documentations for specifics (or experiment in vSNES).
While usually the mode define things like BPP or Max Colors, in SNES each layer has its own definition and is determined by the mode. For example in Mode 0 - all 4 BG are 2 BPP (4 color), but in Mode 1 - 4bpp for BG1,2 and 2bpp for BG3. Sprites (OBJ) are always in 4bpp, regardless of the Mode!

Code: [Select]

$212C - control which layer is visible or not
$2105 - tilesize for each leayer
$2107-$210A - set layer's base address in VRAM and its size. For now we all work with 32x32
 
Now that we know so many things about layers, what exactly we have in a layer?
Each layer is a two dimensional array of 16 bit values. There are 4 possible dimensions, but for now we will talk about 32x32. Each value has this format (exception in Mode 7 or "Offset-per-Tile" modes: 2,4,6):

Code: [Select]

  Bit 0-9     - Tile ID      (000h-3FFh)
  Bit 10-12   - Palette ID   (0-7)
  Bit 13      - BG Priority  (0=Lower, 1=Higher)
  Bit 14      - X-Flip       (0=Normal, 1=Mirror horizontally)
  Bit 15      - Y-Flip       (0=Normal, 1=Mirror vertically)
Clipping???
Scrolling????

[chillyfeez]

Not related to this particular project, but related to this section of your tutorial...
A while ago I was helping Rodimus Primal out by recreating the Japanese FFIV title screen for his "Namingway Edition" hack. The major roadblock that I ran into was transparency. In the Japanese version, the title screen is composed of (at least) two layers of the "FINAL FANTASY IV" lettering and the large crystal image. I was unable to recreate the "flashing" effect of the crystal and the lettering because the top layer seems to be semitransparent.
So if a layer is either transparent or not... How did they do that?!

[Squall]

Working as a software developer, I  learned one thing about the graphic: eye is easily deceived. What you see on the screen could be totally different on how its implemented. For example, when you battle in FF4, you may think that all monsters and chars are sprites. But if you use vSNES you will see that monsters (not sure about chars) are actually in BG1 or 2 (so to speak they are static picture) :D

Unless you do save-state we can only speculate, but if I have to do it, I will probably make the top-most layer tile prerendered of the crystal and the flashing effect (you merge the two layers in Photoshop and the result is assigned to top-most tile).

Speaking of pre-render effect ... in FFBE check the candles - both walls and candles have clear vision, but if you take sometime and watch the fire/light animation over the wall ... see how pixelize it is (effect that happen from rendering the animation in true color(24/32bits) and then converted to less colors, most like 8bit/256). So instead each frame of fire to be rendered in realtime, they use bitmaps with prerendered effect already applied.

[Squall]

III Tiles
We used that word so much, so I hope you got some intuitive understanding, but now is it a time to get a more formal one :D

1. What is a Tile
Tile is the basic element to a tile engines. It is the atom of the chemistry so to speak. It is the brick of an wall :D
A tile is a 2D MxN matrix (block/array) of points/pixels. In SNES a tile is an 8x8 block of pixels!. The colors of each pixels could be an index in a palette or Direct Color (R/G/B values) and that is defined by the Mode/Layer

2. Internal storage (or Bit-planes)
A tile is stored in the memory as a Bit-Planes. Depending on how many BPP is a layer, the tile uses the same amount of Bit-planes. Since SNES can use only 2/4/8 BPP, the tiles uses 2/4/8 Bit-planes. I really, really hope you know what bit-plane is, but if not and you want to know SNES specifics let me know.
From hacking standpoint its important to know - how many bytes a Tile occupy:
- in 2bpp, we have 2 x 8x8 = 128 bits = 16 bytes = $10 bytes
- in 4bpp, we have 4 x 8x8 = 256 bits = 32 bytes = $20 bytes

3. Tile map
A Tille Map is a 2D matrix MxN where each value is a Tile ID. Wait a minute, that looks dangerously similar to the definition of a layer. YES it is, a layer is an instance (custom case) of a tile map. And not just that, a Sprite is another instance of a TileMap.
Because of the way SNES works (Tiles don't hold palette info), the values of this 2D matrix (TileMap) are 16 bits, with the same meaning as layers:

Code: [Select]

Bit 0-9     - Tile ID      (000h-3FFh)
Bit 10-12   - Palette ID   (0-7)
Bit 13      - BG Priority  (0=Lower, 1=Higher)
Bit 14      - X-Flip       (0=Normal, 1=Mirror horizontally)
Bit 15      - Y-Flip       (0=Normal, 1=Mirror vertically)
4. Tile Size
SNES uses 2 sizes for tiles - 8x8 and 16x16. Wait a minute, didn't you say earlier is only 8x8? Yes I did! 16x16 tiles are actually 4 tiles (2x2) stored in a certain order.

5. Important registers (soon)

[chillyfeez]

OK, I have some questions about this section...

1) is it important for me to know specifically what "MxN" means? You used that a couple of times and I don't know what M and N stand for in this instance.

2) So SNES can only do 2/4/8 BPP? What about the 3BPP format that is very prominent in FFIV? Is that just a somewhat-compressed version of 4BPP? I did notice that when copying (0x18 byte) 3BPP tiles to VRAM, the last 8 bytes get spread out over 0x10 bytes. Is that essentially taking the tile data and converting it to 4BPP?

3) I don't know if it's really important for me to know, since there are graphics viewing and editing programs, but I am curious - how exactly is image data encoded? If four bits are used to define a pixel, what do each of those bits represent?
This may also be an appropriate time to mention that I don't really know what a bit plane is...

[Squall]

Excellent questions, m8! Asking questions is the right way of understanding :D

1) MxN usually in the math mean a two dimensional matrix with M rows and N cols. It pretty much mean the same in programming for arrays. The importance is that in it represent rectangular block, but we don't care about exact dimension. If it was MxM or NxN it will mean square rather rectangular. In short MxN stands for 2D rectangular matrix/array/block without specifying the exact dimension.

2) You really nail it!!! Excellent question!!!
As you said FF4 uses internally 3bpp for most monsters and many other graphical objects (inheritance from NES I guess, because FF5 uses 4bpp in most cases). The problem is SNES has 2 or 4 bpp (without loosing quality). That's why all 3bpp stored in the ROM needs to be converted to 4bpp. That is done by inserting a 4th bit-plane with zeroes. 4(decimal) = 100 (binary-3bits) = 0100 (binary-4bits). Because of the way bit-planes are stored, your observation is right: it add 8 zeroes over the last $10 bytes.

Because 3bpp require: 3 x 8x8 bits = 192 bits = 24 bytes = $18, but 4bpp require $20 bytes, we will save 8 bytes per Tile and since we have plenty of Tiles building the image, some people call this compression. FF5 monsters could be 3bpp or 4bpp. There is a flag that tell us that, so in Japanese documents it was named compression :D

3) I don't know if it's really important for me to know, since there are graphics viewing and editing programs

Pretty much NO. Manipulating the individual pixels of a Tile is used mainly for creating special effects. In my observations, Square were a little bit lazy and were doing only effects trough manipulating PPU registers (easy and fast). The other case is for creating Tiles on the fly (trough code) - a library of functions that can manipulate Tiles, like Plot(TileID, X, Y, Color). In FF4, probably the only occasion of manipulating Tiles is the conversion of 3bb -> 4bpp.

[Squall]

III.2. Tile internal storage (Bit-planes) EXTENDED

2.1 What is a Bit-plane
A Bit-Plane is a 2d MxN matrix/array/block of 1 bit values, yes only 0 or 1!
In graphics, if we take bit 0 (rightmost) of each pixel of a picture, the 2d matrix of all bit 0 is a Bit-Plane0. Similarly, of all bit1 of pixels of a picture is Bit-Plane1 and so on.

Lets talk specifically about Tiles in 4BPP:
If we take all bit 0 of all 8x8 pixels, we will have 8x8 1bit matrix - Bit-Plane0
If we take all bit 1 of all 8x8 pixels, we will have 8x8 1bit matrix - Bit-Plane1
If we take all bit 2 of all 8x8 pixels, we will have 8x8 1bit matrix - Bit-Plane2
If we take all bit 3 of all 8x8 pixels, we will have 8x8 1bit matrix - Bit-Plane3
* each Bit-Plane is 8 bytes, each byte contain a row of bits
* Number of BPP = Number of Bit-Planes!

2.2. Example

Code: [Select]

   Tile in Decimal:                     Same Tile in Binary (4bit):
00 01 02 03 04 05 06 07          0000 0001 0010 0011 0100 0101 0110 0111
08 09 10 11 12 13 14 15          1000 1001 1010 1011 1100 1101 1110 1111
00 01 02 03 04 05 06 07          0000 0001 0010 0011 0100 0101 0110 0111   
08 09 10 11 12 13 14 15   ---\  1000 1001 1010 1011 1100 1101 1110 1111
00 01 02 03 04 05 06 07   ---/  0000 0001 0010 0011 0100 0101 0110 0111
08 09 10 11 12 13 14 15          1000 1001 1010 1011 1100 1101 1110 1111
00 01 02 03 04 05 06 07          0000 0001 0010 0011 0100 0101 0110 0111
08 09 10 11 12 13 14 15          1000 1001 1010 1011 1100 1101 1110 1111


Code: [Select]

Bit-Plane 0:        Bit-Plane 1:      Bit-Plane 2:       Bit-Plane 3:
0 1 0 1 0 1 0 1   0 0 1 1 0 0 1 1   0 0 0 0 1 1 1 1   0 0 0 0 0 0 0 0
0 1 0 1 0 1 0 1   0 0 1 1 0 0 1 1   0 0 0 0 1 1 1 1   1 1 1 1 1 1 1 1
0 1 0 1 0 1 0 1   0 0 1 1 0 0 1 1   0 0 0 0 1 1 1 1   0 0 0 0 0 0 0 0
0 1 0 1 0 1 0 1   0 0 1 1 0 0 1 1   0 0 0 0 1 1 1 1   1 1 1 1 1 1 1 1
0 1 0 1 0 1 0 1   0 0 1 1 0 0 1 1   0 0 0 0 1 1 1 1   0 0 0 0 0 0 0 0
0 1 0 1 0 1 0 1   0 0 1 1 0 0 1 1   0 0 0 0 1 1 1 1   1 1 1 1 1 1 1 1
0 1 0 1 0 1 0 1   0 0 1 1 0 0 1 1   0 0 0 0 1 1 1 1   0 0 0 0 0 0 0 0
0 1 0 1 0 1 0 1   0 0 1 1 0 0 1 1   0 0 0 0 1 1 1 1   1 1 1 1 1 1 1 1

2.3 How Bit-Planes are stored in SNES
In the perfect world we will be done, but in SNES things are even more complicated. In memory this is present as:

Code: [Select]

  p -  Bit-Plane,   pX - Bit-Plane X
  r -  row,  rX - row X
  pX:rY - a byte value made by the 8 bits of a row Y in Bit-Plane X
p0:r0  p1:r0  p0:r1  p1:r1  p0:r2  p1:r2  p0:r3  p1:r3  p0:r4  p1:r4  p0:r5  p1:r5  p0:r6  p1:r6  p0:r7  p1:r7
p2:r0  p3:r0  p2:r1  p3:r1  p2:r2  p3:r2  p2:r3  p3:r3  p2:r4  p3:r4  p2:r5  p3:r5  p2:r6  p3:r6  p2:r7  p3:r7

2.4 Fill the missing XXs:
As a practical task, I will give you the value of few pX:rY. Your task is to tell me the missing ones marked as XX. All values are in hex!

Code: [Select]

55 33 XX XX XX XX XX XX XX XX XX XX XX XX XX XX
0F 00 XX XX XX XX XX XX XX XX XX XX XX XX XX XX

2.5 Some interesting cases:
- 2bpp? Its very easy, we have only 2 bit-planes and only 1 row (16 bytes). Its not by chance I used 2 rows to put bytes of a Tile :)
- 3bpp? First row is the same. Second row is only 8 bytes and contain bytes of Bit-Plane 2.
- 3bpp->4bpp? (please tell, how do you think it will happen)