Zahlman's tools Options

[Zahlman]

This is a thread for planning and discussion for Nightmare 3 (as described here) and related tools, and anything else I end up creating that's useful for FE hackers.

So far, we have:

Tools

June 11, 2010: GBA Song Editor v0.30 - not really a proper editor; but this lets you rip and burn songs between ROMs, convert midis, and save the songs in a temporary binary format for further editing. Everything you need to know (I hope) is explained in the README.

August 26, 2010: GBA Song Editor v0.37 is out. I removed the version number from the zip name so that this link will always work for future versions.

Sorry for the delay. This release enhances MIDI support. The new features:

- Certain metadata in MIDIs is now properly skipped (used to cause the converter to fail).
- Strange MIDI note-on and note-off messages are handled better: zero-volume note-ons are treated as note-offs; non-zero volume note-offs add to the warning count but are otherwise treated as normal note-offs.
- Several improvements to BPM handling. The converted song will typically be a bit smaller than before, and timing may be a tiny bit different (but it should never be noticeable, and is still accurate). Also, the BPM value used now follows the same convention as official GBA tracks.
- Converted midis can be set up to loop when you convert them. The loop will connect the end of the song to any specified offset (in "ticks" - 24ths of a beat) from the beginning.

The interface has changed slightly for converting MIDIs. You now specify the instrument offset separately with the 'offset' command, and just provide the song name to 'convert'. There are also commands 'loop' and 'noloop' used to control the looping.

This post has been edited by Zahlman: Aug 26 2010, 04:53 AM

[The Good Guy]

Quick question, how would you get the track offset for any game?
I read the readme.

This post has been edited by The Good Guy: Jun 12 2010, 01:00 AM

[Blazer]

Oooh, seems pretty cool, but I'll wait for the complete program, I think... i.e. when it can do all the insertion and other stuff for me. XD

Oh and good luck with Nightmare 3. It is quite ambitious but I think you, if anyone, can do it.

[shadowofchaos]

Hmm... well it works like a charm my friend...

However, when importing the samples... how much space does it use up? >_>

I'm kinda afraid to use it because of my expanded tables/map sprites ._.

So... the samples... have you figured out why they don't work if they're imported on an expanded ROM? As in on the instrument map they would have a "09" pointer instead of a "08"


Edit: Wait wha!? Your program expands the rom... writes the header and tracks at 1000000... and then the SAMPLES!?

How'd you get it to work!? D:

Edit: On the .bin file that contains everything... would there be any way to tell it where to write in the ROM by changing one of the offsets?

For ripped songs, it's an offset relative to *the start of the file*. (This also allows the burning code to find the end of the last track.)

06 00 00 0A 9C 04 00 00

That's in the .bin file...

06 tracks...

00 00 (according to the readme)

0A Instruments

049C bytes is the location of the instrument map...

Which when I used the program... wrote the instrument map at 0x100049C, followed by the samples...

Is there any way to change that?

[Zahlman]

Quick question, how would you get the track offset for any game?
I read the readme.

By chasing pointers, possibly with the help of Nightmare.

As an example, for FE 7: At offset 0x69D6E0 there is a table of pointers to tracks. Actually, it's a table of 2-word (8-byte) structures: the first word is the track pointer, and the second word indicates the music priority. You can feed these directly to the program, after converting the pointers into offsets (swap bytes and subtract the ROM offset - this should be familiar to y'all by now). However, these are not in the same order as the sound room; also, there are many, many "songs" (keep in mind this includes special effects like e.g. the sound of units moving) that aren't in the sound room at all. If you're interested specifically in those songs, though, you can figure out what maps to which with the help of the sound room data table at offset 0xCE4D28, where the first word of every 4-word structure is an index (not pointer or offset!) into the 0x69D6E0 table, and the last is an index into the text table (which you can view with FEditor).

Blazer has some .nmm files for this over on FEShrine. I hope to compile the information in a more useful way (specifically for these purposes) later, so that you can supply a song index or name and the program will look it up in a list, if provided.

As another example, for Advance Wars: I didn't particularly care what song I inserted for the example, so I just looked for one. First, I found the instrument map region by looking for the word 01 3c 00 00 (a common pattern ). The first hit took me into a region where basically every third word looked like XX 3c 00 00 - perfect; this must be it. I found (by visual inspection) the beginning of this pattern, and then looked for a pointer to the beginning of it. The data around that word looked like song metadata, as expected - the word before is a track count (in the first byte), priority and reverb information, then the instrument map pointer, then pointers to each track (just one in this case). GBA ROMs typically have all the song data packed together (the tracks for a song immediately followed by the metadata connecting them, one after the other for each song). This "song" was really just some special effect, though, so I scrolled through this region of the ROM looking for metadata blocks and tried their offsets until I found something useful for the example: a song with multiple tracks, some but not all of which will burn (I don't quite know how to handle certain types of instrument yet), which actually sounds interesting.

[Obviam]

BwdYeti has provided documentation on which songs have which indices, which are mostly shared between the 3 GBA FEs.

As well, the documentation Zahlman just gave on the structure of the "track table" or whatever you want to call it should be near BwdYeti's provision in my doc.

Yes, the information was available to you already.

0x869D6E0 - Sound pointer table (for music and SFX)
0x869F620 - End of table (0x3E8 entries)

Entries are 8 bytes

First 4 bytes - Sound data pointer
Next 2 bytes - Sound priority - 0 is highest
Next 2 bytes - Copy of previous

0x0000 Nothing
0x0001 Precious Things
0x0002 Companions
0x0003 Friendship and Adventure
0x0004 Dragon's Gate II
0x0005 Scars of the Scouring
0x0006 Raise Your Spirits
0x0007 Shadow Approaches
0x0008 Binding Ties
0xetc. etc.

See?

[Zahlman]

However, when importing the samples... how much space does it use up? >_>

The 'info' command will tell you. Or you can burn at the end of the ROM and see how much bigger it gets. :)

I'm kinda afraid to use it because of my expanded tables/map sprites ._.

Fear of this sort is exactly why you should be using a version control system like I was talking about in the README. :)

So... the samples... have you figured out why they don't work if they're imported on an expanded ROM? As in on the instrument map they would have a "09" pointer instead of a "08"


Edit: Wait wha!? Your program expands the rom... writes the header and tracks at 1000000... and then the SAMPLES!?

How'd you get it to work!? D:

Huh? I didn't realize anyone had any problems with putting samples past the 1000000 mark. I just wrote code that copies all the relevant data, chops it up a bit, and makes all the pointers point where they should.

Actually copying the samples was not hard. They're raw audio data with a 16-byte header that is actually documented in the Sappy docs. There are no pointers to change there, and the header tells me how many bytes to copy.

By the way, this means that if you transfer multiple songs from the same source ROM, you are likely to end up with a lot of duplicated samples. Sorry about that.

Edit: On the .bin file that contains everything... would there be any way to tell it where to write in the ROM by changing one of the offsets?

You tell it where to write in the ROM by giving a second number to the 'burn' command. See 'help burn'.

However, it will still write the whole thing - tracks, instruments and samples. If you want to use an existing map, you can save a binary file, edit it to remove instruments and samples, and then load and burn it. You would want change the instrument count to zero, change the instrument map offset into the actual pointer you want to use, figure out where the instrument map data starts in the binary file, and delete everything from there onward. The program will then look at the data and think it came from converting a MIDI.

06 tracks...

00 00 (according to the readme)

0A Instruments

Yeah, actually I'm not happy about that. I should have (and might change it) put the instrument count in the (normally unused) second byte instead of the last, and kept the priority/reverb information for ripped tracks. Right now your priority/reverb always gets set to 0A 80 when you burn (it seemed to be the most common value for FE 7 songs; at the time, I didn't know what it meant).

Is there any way to change that?

See above.

[Zahlman]

Oooh, seems pretty cool, but I'll wait for the complete program, I think... i.e. when it can do all the insertion and other stuff for me. XD

Did you try it? Aside from setting loop points, I don't really know what you want it to do that it doesn't already, based on our discussions. Although I do have some plans for enhancements to make it easier to use. I might also try doing conversion the other way (to midi), although I would need additional metadata to make that work at all sanely (i.e. a list for each ROM that says what instrument is being played for each sample, so that I can pick a reasonable MIDI instrument ID).

[shadowofchaos]

Hmm... wow... and this is only version 0.30...

Amazing work Zahlman :D

Hmm... so your method would probably work the same way for finding music tables on GBA ROMs Sappy 2006 can't?

My particular ROMs of interested was the Kingdom Hearts: Chain of Memories, and a few Super Robot Wars/Taisen games that Trent asked me to manually import music from ._.

By the way, this means that if you transfer multiple songs from the same source ROM, you are likely to end up with a lot of duplicated samples. Sorry about that.

That's not likely to be a big problem at all... and I don't think it'll be that hard to fix either... but by the time someone will take the time to fix that... I imagine a lot of work has been put into the ROM from expanding it >_>

[Zahlman]

Hmm... so your method would probably work the same way for finding music tables on GBA ROMs Sappy 2006 can't?

Credit for that method goes to Pukachi. Everything I know about GBA music I pieced together from Pukachi's doc, Blazer's doc, and the Sappy doc. I found some other stuff, but none of it was actually helpful. But yes, it should work on any ROM that uses this music engine. Although Sappy really ought to be able to find them in any such ROM, too. No idea why it wouldn't. o_O

(You can also confirm that you're in the right part of the ROM for song data by following the supposed track pointers. Tracks normally start with BC 00 - code for "key transposition: none". I don't think it technically needs to be there, but I guess the folks at Nintendo/IS have an exporter that automatically puts it there. Maybe it reuses the transposition from the previous track if you don't specify one, or something.)

That's not likely to be a big problem at all... and I don't think it'll be that hard to fix either... but by the time someone will take the time to fix that... I imagine a lot of work has been put into the ROM from expanding it >_>

Oh, I plan to have that fixed in the long run. It will be fixed by throwing this toy away and using the appropriate Nightmare 3 modules instead >:)

[Blazer]

Did you try it? Aside from setting loop points, I don't really know what you want it to do that it doesn't already, based on our discussions. Although I do have some plans for enhancements to make it easier to use. I might also try doing conversion the other way (to midi), although I would need additional metadata to make that work at all sanely (i.e. a list for each ROM that says what instrument is being played for each sample, so that I can pick a reasonable MIDI instrument ID).

Well, I mainly needed a program for loop points and repointing. But ultimately this program will be more useful for everyone else who couldn't do music insertion to start with.

GBA2MIDI can already convert GBA songs to MIDI, so you don't really have to make a program for that, but as usual, it's your choice.

For Super Robot Taisen games, I'm almost 100% sure that they don't use the Sappy engine. That's the only reason why I and none of the programs used could find it (yes, I volunteered to search for Seph but couldn't come up with anything). While the format GBA FEs use is the most common, there are other games that don't use it. SRT must be one of those (blame the producers or something, idk).

By the way, I might have just missed this, but did you figure out how to insert WAV samples, or just transfer them from one game to another?

About the first 4 header bytes of a song's header table... I think TT 00 II 00 would work better, like you said, with the 0A (10 instruments) being the 3rd byte instead of the 4th/last byte. Well, it probably doesn't make a difference. The ELF method does it that way though. XP



Another question for you--how exactly does this decide where to put all the data? Will it just put everything after the offsets I give it? If so, that's great, because I'll just give it an offset with like a meg of space after it. If it has to insert data before that space, or somewhere else besides that (excluding the standard MIDI instruments--I already have those inserted anyway), then that could be a problem with my space management. D:

[Obviam]

> GBA2MIDI

Multi platform bro. Python is, .exes aren't. Or did you want to tell me about how GBA2MIDI is either not an 80xecutable or at least built for every major platform?

> Where is my shit going?

Read the source. That is why it is open source.

> I don't know Python.

It's not that hard.

> For me it is!

Diff your ROM then, Jesus.

[Blazer]

^How about everyone just gets Windows, or we forget about the ones who don't? XP. (alternatively Zahlman can go ahead and do it, I just feel bad he can't just like, magically port GBA2MIDI to Python somehow XD)

Xeld. Do you know the point of asking a question? To find out some information that is already known that cannot be quicker found out yourself. It might take Zahlman 5 minutes (if he's a slow typer and types/thinks a lot) to concisely tell me how the data is managed/where it's put. It'd take me at least an hour to get the basics of how his python code works, if not more--it could take several hours. Do a test ROM and figure it out? Might also take 30 minutes or longer, considering I'm not a hacking genius or anything close.

Please don't be so difficult with me Xeld. I just asked a question. And it wasn't even to you. If Zahlman doesn't want to answer it, he can tell me that personally, without bashing me.

If you read this and think "Blazer's a fuckhead who doesn't make any sense and overreacts to me being my normal jerk self" or something even remotely similar, here's the short story: STFU.

This post has been edited by Blazer: Jun 12 2010, 05:04 PM

[Zahlman]

Can we please not have a thread full of fighting about silly things. While problem-solving is a useful and important skill to have, this is a simple program that should not require very much of it. Clearly the README didn't cover everything (they never really do), so I'm here to answer questions. The program is not open-source so that people can figure things out for themselves; it's open-source so that (a) people can trust that it's not a virus or trojan; (b) people can contribute improvements; © I don't have to go to the extra effort of making an executable from Python code. (OK, technically I could just provide .pyc files, but meh.)

Well, I mainly needed a program for loop points and repointing. But ultimately this program will be more useful for everyone else who couldn't do music insertion to start with.

Well, this thing does repointing, so...

By the way, I might have just missed this, but did you figure out how to insert WAV samples, or just transfer them from one game to another?

The header is fully documented in Sappy's doc:

CODE

Samples
Type Name Description
u16 type Not used
u16 stat 0x0000 for oneshot, 0x4000 for forward loop
u32 freq "Normal" frequency << 10
u32 loop Loop start position in samples
u32 size Total number of samples
s8 data[] Size+1 samples in total.
Last byte is zero for one-shots, same as loop pointer for looped samples.

The sample data is then padded for word alignment. As 's8' indicates, each sample is 8 bits, signed. The samples are mono (not stereo). I don't really know how the "normal" frequency works, but from some quick testing and fooling around*, it seems to be the sample frequency that makes the sample play a middle C. I'll have to test this more.

* I copied a sample to a new file, manually created a WAV header for it, and tried various sample rates - did a little math, hummed some notes to compare to the sample, yadda yadda. Not as exact a science as I'd like.

About the first 4 header bytes of a song's header table... I think TT 00 II 00 would work better, like you said, with the 0A (10 instruments) being the 3rd byte instead of the 4th/last byte. Well, it probably doesn't make a difference. The ELF method does it that way though. XP

Except that the existing data is TT 00 PP RR, so I should write TT II PP RR in order not to lose the priority and reverb information. That was the point.

Another question for you--how exactly does this decide where to put all the data? Will it just put everything after the offsets I give it?

Everything is written in a single chunk, yes. You an use the 'info' command to see how much space is needed. The info output says "will need <X> contiguous bytes" for a reason. ;)

This post has been edited by Zahlman: Jun 12 2010, 10:11 PM

[Obviam]

Being implored to try something faster for everyone is no excuse to throw a hissy fit. It'd have been fine except yes, you did overreact instead of calmly stating your displeasure.

AKA that last sentence wasn't necessary.

No one's directly talking to you either; everyone has "where is it going?" questions since ever and someone like you and most of the others that ask that question are perfectly capable of looking themselves.

[Spectre]

blazers in here getting all pissy

and egberts here getting it busy

guy doesnt know how to ask a damn question

i think this calls for a little sequestration

in this here bro-nation

cause blazer just got schoolfed

homestead

like the fuckin jesus said

shit be nasty yo

motherfuckin OWNED

[Blazer]

@Zahlman, alright. I was a bit stupid there. Thanks for the answers, I'll be sure to make use of this program. Oh, and great work, man. :P

[Obviam]

sir, I'm going to have to ask you to display your rapping license. these are public premises.

[Spectre]

oh shit, officer, i think i broke the law

i mean, these rhymes are just too sick

(Breaking character to be on topic, holy shit zahlman awesome.)

[Xenesis]

Holy crap.

This is awesome. I'm going to spend some time with it tonight seeing if I can get my song stuff working better! :)

Midi conversion was always a pain in the arse and anything that potentially makes it easier sounds good in my book.