d2jsp
Log InRegister
d2jsp Forums > Off-Topic > Computers & IT > Programming & Development > Java Printing To Excel
12Next
Add Reply New Topic New Poll
Member
Posts: 5,269
Joined: Oct 18 2006
Gold: 21,400.00
Jul 10 2013 12:23pm
So i have been using the apache POI library to write to excel from java for awhile now. My typical approach is to create the entire workbook. Then create a FileOutputStream. Then write the OutputStream to the workbook. Then flush and close the OutputStream.

Code
/* Creating the Workbook/sheet/row/cells... */

FileOutputStream excelOut = new FileOutputStream(EXCEL_FILE_OUT);
inventoryWorkbook.write(excelOut);
excelOut.flush();
excelOut.close();


This approach works great on small and medium size projects, but now I am doing a massive excel file dump (50k+ lines). Before being 10% done I get a "Exception in thread "main" java.lang.OutOfMemoryError: Java heap space"

What is the conventional way to periodically write to the excel file and clear up heap space?
Member
Posts: 4,554
Joined: Dec 1 2008
Gold: 0.50
Jul 10 2013 12:27pm
I'd store it in an SQL db then import to Excel
Member
Posts: 5,269
Joined: Oct 18 2006
Gold: 21,400.00
Jul 10 2013 12:32pm
Quote (Furio @ Jul 10 2013 11:27am)
I'd store it in an SQL db then import to Excel

My program is reading from a DB, manipulating the data, then exporting into Excel.
Member
Posts: 32,925
Joined: Jul 23 2006
Gold: 3,804.50
Jul 10 2013 05:29pm
i've always built excel via XML instead of using those objects. or is that what the poi library does?

if the problem occurs when writing the stream, use a buffered stream writer to only write a little bit at a time. but if it says heap space, then i think you're creating too many objects. look for whatever loops you're using. perhaps you can improve it. alternatively, if this is in xml, you might have to write to disk more frequently to clear that data from memory. eg write 10,000 (or whatever) rows at a time to disk, then free the memory and continue

or why are you writing your own program to do it anyway? is it possible to just create a result set from the manipulation and export that in your DBMS?

This post was edited by carteblanche on Jul 10 2013 05:30pm
Member
Posts: 3,451
Joined: Feb 26 2010
Gold: 0.20
Jul 12 2013 04:53am
seems like your'e running out of heap space
Member
Posts: 5,269
Joined: Oct 18 2006
Gold: 21,400.00
Jul 12 2013 10:51pm
Quote (carteblanche @ Jul 10 2013 04:29pm)
i've always built excel via XML instead of using those objects. or is that what the poi library does?

if the problem occurs when writing the stream, use a buffered stream writer to only write a little bit at a time. but if it says heap space, then i think you're creating too many objects. look for whatever loops you're using. perhaps you can improve it. alternatively, if this is in xml, you might have to write to disk more frequently to clear that data from memory. eg write 10,000 (or whatever) rows at a time  to disk, then free the memory and continue

or why are you writing your own program to do it anyway? is it possible to just create a result set from the manipulation and export that in your DBMS?


The POI library uses objects which is lame because of the heap space and more importantly, you can only use versions 2003 and lower.

Does using XML not require building objects, but just printing directing to the file (almost like Excel.out.print(<cell>asdf</cell>))? If so that would be perfect for me. I also did some more calculations and my program will write ~200k lines which is too long for a excel 2003 table.

I'm not quite sure about your last question. My end product needs to be an excel file, not part of a db.
Member
Posts: 32,925
Joined: Jul 23 2006
Gold: 3,804.50
Jul 12 2013 11:07pm
it's just xml. build it any way you want to.
http://blogs.msdn.com/b/brian_jones/archive/2005/06/27/433152.aspx

i imagine oracle, sql server, postgres, etc all have an IDE/DBMS with an export feature. sql workbench (google it if you need) is a free jdbc opensource IDE for sql. given a result set from a select statement, you can save it as excel xml. just rename to xls and open it up in excel.

This post was edited by carteblanche on Jul 12 2013 11:08pm
Member
Posts: 5,269
Joined: Oct 18 2006
Gold: 21,400.00
Jul 12 2013 11:21pm
Quote (carteblanche @ Jul 12 2013 10:07pm)
it's just xml. build it any way you want to.
http://blogs.msdn.com/b/brian%5Fjones/archive/2005/06/27/433152.aspx

i imagine oracle, sql server, postgres, etc all have an IDE/DBMS with an export feature. sql workbench (google it if you need) is a free jdbc opensource IDE for sql. given a result set from a select statement, you can save it as excel xml. just rename to xls and open it up in excel.


Thanks for the link, this is the direction I will take. Quick question though:
Using his simple example:
Code
<?xml version="1.0"?>
<ss:Workbook xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet">
   <ss:Worksheet ss:Name="Sheet1">
       <ss:Table>
           <ss:Column ss:Width="80"/>
          <ss:Column ss:Width="80"/>
          <ss:Column ss:Width="80"/>
           <ss:Row>
               <ss:Cell>
                   <ss:Data ss:Type="String">First Name</ss:Data>
               </ss:Cell>
               <ss:Cell>
                   <ss:Data ss:Type="String">Last Name</ss:Data>
               </ss:Cell>
               <ss:Cell>
                   <ss:Data ss:Type="String">Phone Number</ss:Data>
               </ss:Cell>
           </ss:Row>
       </ss:Table>
   </ss:Worksheet>
</ss:Workbook>

If I save that with Notepad++ as test.xlsx and try to open it in Excel 2007 I get an error saying bad formatting. What am I doing wrong?
Member
Posts: 32,925
Joined: Jul 23 2006
Gold: 3,804.50
Jul 13 2013 09:38am
did you try saving it as .xls? i dont have excel on my home computer so i can't test it out. on my work computer i have excel 2003. in excel 2003, build an excel sheet and go to file save as, and one of the options under the extensions is "xml workbook xls" or something like that. i assume 2007 also has that. looks for the xml options, then open it up again in notepad++ to test it
Member
Posts: 5,269
Joined: Oct 18 2006
Gold: 21,400.00
Jul 13 2013 09:24pm
yeah saving the file as .xls worked and it had 1m+ rows so it was a valid 2007+ document. Thanks for the help, I really appreciate it!
Go Back To Programming & Development Topic List
12Next
Add Reply New Topic New Poll