How do I open a 2.5 gig .xml file?

[atorch]

Hi,

This is going to sound a little silly, but... I have an .xml file I want to open, and I can't. It's very big (2.5 gigs), and I imagine that's part of the problem.

Firefox wants to open it by default, but it only takes me to a blank page, exactly the same as about:blank.

Text Editor tells it to me like it is: it cannot open the file because it is "too big."

OpenOffice Database just eats up tons of RAM when it tries to open the file...

I tried to open it with Notepad (through Wine), but it didn't do anything at all. Didn't eat up any memory, didn't open the file...



What can I do? I need to either i) open this file or ii) convert it into something that I can open.

[Oldsoldier2003]

Originally Posted by atorch Hi, This is going to sound a little silly, but... I have an .xml file I want to open, and I can't. It's very big (2.5 gigs), and I imagine that's part of the problem. Firefox wants to open it by default, but it only takes me to a blank page, exactly the same as about:blank. Text Editor tells it to me like it is: it cannot open the file because it is "too big." OpenOffice Database just eats up tons of RAM when it tries to open the file... I tried to open it with Notepad (through Wine), but it didn't do anything at all. Didn't eat up any memory, didn't open the file... What can I do? I need to either i) open this file or ii) convert it into something that I can open.

Depending on what you want to do with it... if its a repetitive edit you could just use SED and make replacements without ever "opening the file". If its something you have to be able to read you might be better off splitting the file into usable chunks.

[atorch]

Originally Posted by Oldsoldier2003 If its something you have to be able to read you might be better off splitting the file into usable chunks.

Yes, I have to be able to read it. How do I go about splitting the file? (I didn't create it myself -- it's data downloaded from a site whose entire history is kept in one big .xml file.)

Edit: Is there any way to lift Text Editor's file size limit?

[jaakan]

you need to import that in to a database like mysql or postgresql and just run queries against them.

[lavinog]

what happens if you use head?

head myxmlfile.xml

(do this in a terminal)

or does this work?

 less myxmlfile.xml

gedit?

[bruce89]

Jings, is that the OpenStreetMap planet file? Can't be, it's 3.7 GB compressed with bzip.

[atorch]

Originally Posted by jaakan you need to import that in to a database like mysql or postgresql and just run queries against them.

Sorry to be a total nublet, but how do I do that?

Google has led me here:
http://codeghar.wordpress.com/2007/1...-gutsy-gibbon/

So I ran these commands:
sudo apt-get install mysql-server
sudo apt-get install mysql-client
sudo apt-get install mysql-admin
sudo apt-get install mysql-query-browser

Now under Applications > Programming I find:
MySQL Administrator
MySQL Query Browser

But both those applications seem to be made to connect to databases on other computers... how do I use them to open an .xml file on my own desktop? (If it is at all possible to do so...)

Edit: Or, how do I convert .xml into a format the MySQL works with?

(As you can probably tell, I have at best a very very vague idea of what MySQL is.)

[atorch]

Originally Posted by lavinog what happens if you use head? head myxmlfile.xml (do this in a terminal) or does this work? less myxmlfile.xml gedit?

head
yes, but it seems to be opening only the the first few lines of the file. (The header?)

less
yes, this works, and I can use the down arrow to move through the file.

the problem with both head and less is that they open the file without any formatting... but at least I know the file opens and isn't corrupted or something

gedit does *not* work, it tells me the file is too big, just like Text Editor did. in fact, gedit seems to be the same as Text Editor

[jflaker]

MySql 6.0 seems to have an implementation of Load XML......based on Load Data..

http://dev.mysql.com/doc/refman/6.0/en/load-xml.html

I may be worth playing with the new version just to get your data into a DB so you can do something useful with it. There is no mention of max file size for the XML

Good luck

[lavinog]

Originally Posted by atorch head yes, but it seems to be opening only the the first few lines of the file. (The header?) less yes, this works, and I can use the down arrow to move through the file. the problem with both head and less is that they open the file without any formatting... but at least I know the file opens and isn't corrupted or something gedit does *not* work, it tells me the file is too big, just like Text Editor did. in fact, gedit seems to be the same as Text Editor

Well what information are you trying to view?
less gives you the same output notepad would.
I also don't think firefox makes it much better.

it looks like tools such as grep and sed would also work
you can use:

split -l 5000 myxmlfile.xml newxml

this would split up the xml file into smaller files
you can change 5000 to whatever number of lines you need
there is a way to split it up at every occurance of a certain word using csplit