Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: python script request - reading from *.csv, writing to files

  1. #1
    Join Date
    Jan 2011
    Beans
    41

    python script request - reading from *.csv, writing to files

    I have spreadsheets for modifying data structures in old video games, which, when edited, concatenate all the edited data together into another worksheet, which is then saved as a *.csv (comma separated value).

    I need a cross-platform script that will grab the data from the *.csv, and write it to the appropriate files, provided those files are in the same directory as the python script and the *.csv.

    The .csv files look like this:
    Code:
    (filename),(address, in hexadecimal),(bytes to write, in hexadecimal)
    SCUS_942.30,9EE0,300001C08000140000000000FFFFFFFF6581FFFFFFFFFFFFFFFFFFFF
    SCUS_942.30,9EFC,300001C08000280000000000FFFFFFFF6581FFFFFFFFFFFFFFFFFFFF
    SCUS_942.30,9F18,300001C08000640000000000FFFFFFFF6581FFFFFFFFFFFFFFFFFFFF
    SCUS_942.30,9F34,300001C08000140000000000FFFFFFFFF481FFFFFFFFFFFFFFFFFFFF
    SCUS_942.30,9F50,300001C08000000000000000FFFFFFFF6981FFFFFFFFFFFFFFFFFFFF
    SCUS_942.30,9F6C,300001C08000010000000000FFFFFFFF6881FFFFFFFFFFFFFFFFFFFF
    As there will probably be multiple *.csv files for editing different data structures in these games, I need the python script to be able to select the *.csv to use, when using the python script in terminal.
    Code:
    python csv_writer.py saga_item_data.csv
    I don't need to select the file(s) to write it to, as they are part of the *.csv; and because some of my spreadsheets work with hundreds of files, and selecting the file to which to write would only hinder me.

    The *.csv files are simply plain text.
    - Probably UTF-8 on Linux, (using Gnumeric or Calc) and ISO-8859-1* on Windows(using Excel).
    - It's probably okay to use Windows CR+LF (which is the Windows line-break character), but don't quote me on that.

    I'm not a programmer; just a data diver who knows how to search for things using a hex editor.

  2. #2
    Join Date
    Apr 2007
    Location
    NorCal
    Beans
    1,149
    Distro
    Ubuntu 10.04 Lucid Lynx

    Re: python script request - reading from *.csv, writing to files

    Generally we don't take program requests here, but help people who are trying to write their own. So, today you're going to learn some python.

    Reading in CSVs:
    http://docs.python.org/library/csv.html

    Converting hex to int:
    http://docs.python.org/library/functions.html#int (base=16)

    Jumping to a position in a file:
    http://docs.python.org/library/stdty...l#file-objects (you want "seek" to find byte x of the file)

    Converting that hex ascii string to a byte string:
    http://code.activestate.com/recipes/...ng-conversion/

    Then write out the byte string to the file.
    Last edited by schauerlich; September 5th, 2012 at 08:37 AM.
    Posting code? Use the [code] or [php] tags.
    I don't care, I'm still free. You can't take the sky from me.

  3. #3
    Join Date
    Apr 2007
    Location
    NorCal
    Beans
    1,149
    Distro
    Ubuntu 10.04 Lucid Lynx

    Re: python script request - reading from *.csv, writing to files

    Lucky for you I was sufficiently bored.

    Code:
    $ cat blah.csv
    bar.bin,0F,313233
    $ cat bar.bin
    abcdefghijklmnopqrstuvwxyz
    $ python foo.py blah.csv 
    $ cat bar.bin
    abcdefghijklmno123stuvwxyz
    $ cat foo.py
    import csv
    import sys
    
    def hex_to_bytes(s):
        return "".join(chr(int(s[i:i+2], 16)) for i in range(0, len(s), 2))
    
    if __name__ == "__main__":
        csv_file = sys.argv[1]
    
        with open(csv_file, "r") as inf:
            reader = csv.reader(inf)
    
            for filename, addr, data in reader:
                addr = int(addr, 16)
                data = hex_to_bytes(data)
    
                with open(filename, "r+b") as outf:
                    outf.seek(addr)
                    outf.write(data)
    extending that to multiple CSVs is left as an exercise to the reader. (hint: it's as simple as deleting one line and adding another)
    Last edited by schauerlich; September 5th, 2012 at 08:44 AM.
    Posting code? Use the [code] or [php] tags.
    I don't care, I'm still free. You can't take the sky from me.

  4. #4
    Join Date
    Jan 2011
    Beans
    41

    Re: python script request - reading from *.csv, writing to files

    Quote Originally Posted by schauerlich View Post
    Lucky for you I was sufficiently bored.

    Code:
    $ cat blah.csv
    bar.bin,0F,313233
    
    $ cat bar.bin
    abcdefghijklmnopqrstuvwxyz
    
    $ python foo.py blah.csv 
    
    $ cat bar.bin
    abcdefghijklmno123stuvwxyz
    
    $ cat foo.py
    import csv
    import sys
    
    def hex_to_bytes(s):
        return "".join(chr(int(s[i:i+2], 16)) for i in range(0, len(s), 2))
    
    if __name__ == "__main__":
        csv_file = sys.argv[1]
    
        with open(csv_file, "r") as inf:
            reader = csv.reader(inf)
    
            for filename, addr, data in reader:
                addr = int(addr, 16)
                data = hex_to_bytes(data)
    
                with open(filename, "r+b") as outf:
                    outf.seek(addr)
                    outf.write(data)
    Extending that to multiple CSVs is left as an exercise to the reader. (hint: it's as simple as deleting one line and adding another)
    I don't need it to work on multiple CSVs (with or without wildcards), or to work on all CSVs with a directory. Doing so may create problems, if some CSVs share the same data, or if someone forgets to remove an out-dated CSV.

    Thank you. Would you mind if I credited you? Do you have any website you'd like me to link to, if someone wants to post a thank-you message, or anything you want advertised?

    EDIT

    My spreadsheets originally were going to use hex editors, but I couldn't find a hex editor that worked correctly - they would insert data instead of just overwriting data (which was what I needed, overwriting). I mentioned this to the author of one such hex editor, and I think he's working on the issue.

    EDIT
    And what version of python is this for?
    Last edited by wheel_walker; September 5th, 2012 at 10:12 PM.

  5. #5
    Join Date
    Feb 2009
    Beans
    1,468

    Re: python script request - reading from *.csv, writing to files

    I'm a little puzzled by why you'd use spreadsheets to begin with. Is using a hex editor to edit the binary directly not an option? Just wondering.

  6. #6
    Join Date
    Jan 2011
    Beans
    41

    Re: python script request - reading from *.csv, writing to files

    Quote Originally Posted by trent.josephsen View Post
    I'm a little puzzled by why you'd use spreadsheets to begin with. Is using a hex editor to edit the binary directly not an option? Just wondering.
    Editing with just a hex editor is perfectly fine, but it gets tedious when you're editing a table with 256 entries, and 27 bytes per entry. Or when you're adjusting a set of pointer tables, and correcting the entries they point to.

    My goal is to make modding these games accessible to everyone, and spreadsheets do a pretty good job of that, especially when combined with the python script here.

  7. #7
    Join Date
    Feb 2009
    Beans
    1,468

    Re: python script request - reading from *.csv, writing to files

    Ah, I see.

    In that case, not to tell you your business, but I would add a field to the spreadsheet for annotations describing each segment -- that would do wonders for accessibility.

  8. #8
    Join Date
    Jan 2011
    Beans
    41

    Re: python script request - reading from *.csv, writing to files

    Quote Originally Posted by trent.josephsen View Post
    Ah, I see.

    In that case, not to tell you your business,...
    Feel free to tell me anything. I like opportunities to learn new things which are easily applied.

    Quote Originally Posted by trent.josephsen View Post
    ... but I would add a field to the spreadsheet for annotations describing each segment -- that would do wonders for accessibility.
    Are you talking about detailed descriptions for each offset, and suchlike? Here's a good example of what I do:
    http://biolab.warsworldnews.com/viewtopic.php?f=5&t=13

    If you're talking about something else entirely, then please tell me more about it.

  9. #9
    Join Date
    Feb 2009
    Beans
    1,468

    Re: python script request - reading from *.csv, writing to files

    I was referring to keeping the annotations in-line with the content. I used a hex editor once a long time ago (last century) that basically read in and interpreted the file for you (using plugins for different kinds of binary files), and allowed you to change the content by setting values in a form with human-readable labels. It would be easy to do that in a spreadsheet format, not entirely different from your Items worksheet. As it is I'm not 100% sure what the rows and columns in that sheet represent -- maybe there's not a good way to improve it.

  10. #10
    Join Date
    Apr 2007
    Location
    NorCal
    Beans
    1,149
    Distro
    Ubuntu 10.04 Lucid Lynx

    Re: python script request - reading from *.csv, writing to files

    Quote Originally Posted by wheel_walker View Post
    I don't need it to work on multiple CSVs (with or without wildcards), or to work on all CSVs with a directory. Doing so may create problems, if some CSVs share the same data, or if someone forgets to remove an out-dated CSV.

    Thank you. Would you mind if I credited you? Do you have any website you'd like me to link to, if someone wants to post a thank-you message, or anything you want advertised?

    EDIT

    My spreadsheets originally were going to use hex editors, but I couldn't find a hex editor that worked correctly - they would insert data instead of just overwriting data (which was what I needed, overwriting). I mentioned this to the author of one such hex editor, and I think he's working on the issue.

    EDIT
    And what version of python is this for?
    Feel free to steal it shamelessly. Or link to this thread. Whatever works for you.

    It should work on anything 2.6+. I believe the with statement was added then.
    Posting code? Use the [code] or [php] tags.
    I don't care, I'm still free. You can't take the sky from me.

Page 1 of 2 12 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •