python script request - reading from *.csv, writing to files

[wheel_walker]

I have spreadsheets for modifying data structures in old video games, which, when edited, concatenate all the edited data together into another worksheet, which is then saved as a *.csv (comma separated value).

I need a cross-platform script that will grab the data from the *.csv, and write it to the appropriate files, provided those files are in the same directory as the python script and the *.csv.

The .csv files look like this:

(filename),(address, in hexadecimal),(bytes to write, in hexadecimal)
SCUS_942.30,9EE0,300001C08000140000000000FFFFFFFF6581FFFFFFFFFFFFFFFFFFFF
SCUS_942.30,9EFC,300001C08000280000000000FFFFFFFF6581FFFFFFFFFFFFFFFFFFFF
SCUS_942.30,9F18,300001C08000640000000000FFFFFFFF6581FFFFFFFFFFFFFFFFFFFF
SCUS_942.30,9F34,300001C08000140000000000FFFFFFFFF481FFFFFFFFFFFFFFFFFFFF
SCUS_942.30,9F50,300001C08000000000000000FFFFFFFF6981FFFFFFFFFFFFFFFFFFFF
SCUS_942.30,9F6C,300001C08000010000000000FFFFFFFF6881FFFFFFFFFFFFFFFFFFFF

As there will probably be multiple *.csv files for editing different data structures in these games, I need the python script to be able to select the *.csv to use, when using the python script in terminal.

python csv_writer.py saga_item_data.csv

I don't need to select the file(s) to write it to, as they are part of the *.csv; and because some of my spreadsheets work with hundreds of files, and selecting the file to which to write would only hinder me.

The *.csv files are simply plain text.
- Probably UTF-8 on Linux, (using Gnumeric or Calc) and ISO-8859-1* on Windows(using Excel).
- It's probably okay to use Windows CR+LF (which is the Windows line-break character), but don't quote me on that.

I'm not a programmer; just a data diver who knows how to search for things using a hex editor.

[schauerlich]

Generally we don't take program requests here, but help people who are trying to write their own. So, today you're going to learn some python.

Reading in CSVs:
http://docs.python.org/library/csv.html

Converting hex to int:
http://docs.python.org/library/functions.html#int (base=16)

Jumping to a position in a file:
http://docs.python.org/library/stdty...l#file-objects (you want "seek" to find byte x of the file)

Converting that hex ascii string to a byte string:
http://code.activestate.com/recipes/...ng-conversion/

Then write out the byte string to the file.

[schauerlich]

Lucky for you I was sufficiently bored.

$ cat blah.csv
bar.bin,0F,313233
$ cat bar.bin
abcdefghijklmnopqrstuvwxyz
$ python foo.py blah.csv 
$ cat bar.bin
abcdefghijklmno123stuvwxyz
$ cat foo.py
import csv
import sys

def hex_to_bytes(s):
    return "".join(chr(int(s[i:i+2], 16)) for i in range(0, len(s), 2))

if __name__ == "__main__":
    csv_file = sys.argv[1]

    with open(csv_file, "r") as inf:
        reader = csv.reader(inf)

        for filename, addr, data in reader:
            addr = int(addr, 16)
            data = hex_to_bytes(data)

            with open(filename, "r+b") as outf:
                outf.seek(addr)
                outf.write(data)

extending that to multiple CSVs is left as an exercise to the reader. (hint: it's as simple as deleting one line and adding another)

[wheel_walker]

Lucky for you I was sufficiently bored.

$ cat blah.csv
bar.bin,0F,313233

$ cat bar.bin
abcdefghijklmnopqrstuvwxyz

$ python foo.py blah.csv 

$ cat bar.bin
abcdefghijklmno123stuvwxyz

$ cat foo.py
import csv
import sys

def hex_to_bytes(s):
    return "".join(chr(int(s[i:i+2], 16)) for i in range(0, len(s), 2))

if __name__ == "__main__":
    csv_file = sys.argv[1]

    with open(csv_file, "r") as inf:
        reader = csv.reader(inf)

        for filename, addr, data in reader:
            addr = int(addr, 16)
            data = hex_to_bytes(data)

            with open(filename, "r+b") as outf:
                outf.seek(addr)
                outf.write(data)

Extending that to multiple CSVs is left as an exercise to the reader. (hint: it's as simple as deleting one line and adding another)

I don't need it to work on multiple CSVs (with or without wildcards), or to work on all CSVs with a directory. Doing so may create problems, if some CSVs share the same data, or if someone forgets to remove an out-dated CSV.

Thank you. Would you mind if I credited you? Do you have any website you'd like me to link to, if someone wants to post a thank-you message, or anything you want advertised?

EDIT

My spreadsheets originally were going to use hex editors, but I couldn't find a hex editor that worked correctly - they would insert data instead of just overwriting data (which was what I needed, overwriting). I mentioned this to the author of one such hex editor, and I think he's working on the issue.

EDIT
And what version of python is this for?

[trent.josephsen]

I'm a little puzzled by why you'd use spreadsheets to begin with. Is using a hex editor to edit the binary directly not an option? Just wondering.

[wheel_walker]

I'm a little puzzled by why you'd use spreadsheets to begin with. Is using a hex editor to edit the binary directly not an option? Just wondering.

Editing with just a hex editor is perfectly fine, but it gets tedious when you're editing a table with 256 entries, and 27 bytes per entry. Or when you're adjusting a set of pointer tables, and correcting the entries they point to.

My goal is to make modding these games accessible to everyone, and spreadsheets do a pretty good job of that, especially when combined with the python script here.

[trent.josephsen]

Ah, I see.

In that case, not to tell you your business, but I would add a field to the spreadsheet for annotations describing each segment -- that would do wonders for accessibility.

[wheel_walker]

Ah, I see.

In that case, not to tell you your business,...

Feel free to tell me anything. I like opportunities to learn new things which are easily applied.

... but I would add a field to the spreadsheet for annotations describing each segment -- that would do wonders for accessibility.

Are you talking about detailed descriptions for each offset, and suchlike? Here's a good example of what I do:
http://biolab.warsworldnews.com/viewtopic.php?f=5&t=13

If you're talking about something else entirely, then please tell me more about it.

[trent.josephsen]

I was referring to keeping the annotations in-line with the content. I used a hex editor once a long time ago (last century) that basically read in and interpreted the file for you (using plugins for different kinds of binary files), and allowed you to change the content by setting values in a form with human-readable labels. It would be easy to do that in a spreadsheet format, not entirely different from your Items worksheet. As it is I'm not 100% sure what the rows and columns in that sheet represent -- maybe there's not a good way to improve it.

[schauerlich]

I don't need it to work on multiple CSVs (with or without wildcards), or to work on all CSVs with a directory. Doing so may create problems, if some CSVs share the same data, or if someone forgets to remove an out-dated CSV.

Thank you. Would you mind if I credited you? Do you have any website you'd like me to link to, if someone wants to post a thank-you message, or anything you want advertised?

EDIT

My spreadsheets originally were going to use hex editors, but I couldn't find a hex editor that worked correctly - they would insert data instead of just overwriting data (which was what I needed, overwriting). I mentioned this to the author of one such hex editor, and I think he's working on the issue.

EDIT
And what version of python is this for?

Feel free to steal it shamelessly. Or link to this thread. Whatever works for you.

It should work on anything 2.6+. I believe the with statement was added then.