Page 2 of 2 FirstFirst 12
Results 11 to 16 of 16

Thread: [Python] Unsigned 64-bit integer causing addressing error

  1. #11

    Re: [Python] Unsigned 64-bit integer causing addressing error

    Well, there you go. 4194304*2*256 = 2^31, and since long is only 32 bits on Windows, you overflow when passing it to seek (file.seek is just a wrapper to C's fseek(), which takes a long). I bet that even with the signed version that doesn't give an error message, you find that you're actually accessing the wrong bytes for newpos > 4194303.

    So the question is... why doesn't it warn you for the signed version? I'm not familiar enough with Python internals to answer that question, although I have a theory that could explain it, but it looks from here like you might have found a bug, either in file.seek() (failing to do the proper conversion) or in numpy (returning the number in a form that confuses Python).

    So what can you do for now? Split the single seek() into several steps, using the optional second argument to make the offset relative to the current file position. That is,

    Code:
        def goto(self, newpos):
            offset = newpos*2*self.rec_len
            whence = 0 # use this to make the first seek() absolute
                       # and the rest relative to the current position
            while offset >= 2**31:
                self.f.seek(2**31 - 1, whence)
                whence = 1
                offset -= 2**31 - 1
            self.f.seek(offset, whence)
    Code for demonstration purposes only, untested, probably has off-by-one errors, etc.

  2. #12
    Join Date
    Jun 2007
    Location
    The Fiery Desert
    Beans
    141
    Distro
    Ubuntu 7.04 Feisty Fawn

    Re: [Python] Unsigned 64-bit integer causing addressing error

    Ho-hum. So if there is an overflow that somehow goes silent, it should just start reading from the beginning of the file again, right?
    Die writing: I write crazy things
    PhD Wander: I write crazy things about my travels
    RedScout: Lenovo 3000 C100, 1.5GHz Celeron M, 1.25GB RAM, 120GB HDD

  3. #13

    Re: [Python] Unsigned 64-bit integer causing addressing error

    Welllll... maybe. This isn't a "standard" integer overflow; you're going from a Python "int" to a C long and there may be several points at which the value won't fit in the container. It could in fact access random bytes, or always access byte 0... there are lots of possibilities, actually, and I don't know enough about Python internals, like I said before, to guess at which one.

    Merry Christmas, anyway.

  4. #14
    Join Date
    Jun 2007
    Location
    The Fiery Desert
    Beans
    141
    Distro
    Ubuntu 7.04 Feisty Fawn

    Re: [Python] Unsigned 64-bit integer causing addressing error

    I haven't noticed anything especially odd going on with the data coming from the tail of the file. Nothing like garbage of flat-lining. At worst, it probably just starts reading at the beginning. But I will have to check more closely when I get back to the work machine.

    Do you think this is worth submitting a bug report to project? I've never done that before.

    Thanks for your help, and Merry Christmas to you, too!
    Die writing: I write crazy things
    PhD Wander: I write crazy things about my travels
    RedScout: Lenovo 3000 C100, 1.5GHz Celeron M, 1.25GB RAM, 120GB HDD

  5. #15

    Re: [Python] Unsigned 64-bit integer causing addressing error

    I'd at least try to find a mailing list or forum relevant to numpy and see if it could be specific to fromfile(). There must be somebody who can tell if it's actually a bug in Python, and if it is, how to replicate it without using numpy.

    This is an specific enough set of circumstances that I wouldn't be surprised to find you're the first one to discover this bug.

    P.S. To be clear, starting over at the beginning of the file is also a likely possibility. But I wouldn't bet money on it, is all I was saying before.

  6. #16
    Join Date
    Jun 2007
    Location
    The Fiery Desert
    Beans
    141
    Distro
    Ubuntu 7.04 Feisty Fawn

    Re: [Python] Unsigned 64-bit integer causing addressing error

    So I wanted to check if it was indeed reading garbage bytes or out of place bytes if I was using unsigned 64-bit integers for specifying the address.

    First, I parked the pointer at the last record that can be accessed safely, then used the .read() class method to advance one record at a time. This simply reads a single record, and then leaves the pointer at the beginning of the next. This way I was able to grab a few records beyond the failure point.

    Then I made a short numpy array consisted of addresses around the failure point, with dtype np.int64 (signed 64-bit integers). Used this array to access specific records, and then compared them to the ones retrieved from the same positions using the "safe" method. The records retrieved using the two methods were identical. Which tells me that using signed 64-bit integers addresses everything correctly. It's only the unsigned 64-bit integers that fail.

    I am going to submit this to numpy's bug tracker. I've never done that before. That's like... extra nerd cred!

    Submitted on github.
    Last edited by Erdaron; January 1st, 2013 at 11:33 PM. Reason: Added bug track link
    Die writing: I write crazy things
    PhD Wander: I write crazy things about my travels
    RedScout: Lenovo 3000 C100, 1.5GHz Celeron M, 1.25GB RAM, 120GB HDD

Page 2 of 2 FirstFirst 12

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •