replaceAll, OutOfMemory

[raylu]

http://java.sun.com/j2se/1.4.2/docs/api/java/lang/String.html#replaceAll(java.lang.String,%20java.lang.String)
I want to recursively convert files in a directory with a UNIX line terminator to the Windows one. I have written a small Java program that does this perfectly using replaceAll. However, on a certain file larger than 8MB, it fails to do this because of an OutOfMemory exception.

What is the limit for using replaceAll and is there a way around this?

[MyndFyre]

Have you considered using streams instead of strings?

[Banana fanna fo fanna]

Ugh you are doing it all wrong. You need an Enterprise String Buffer (ESB) to scale to those types of files. I have out of the kindness of my heart provided one.

Code Select Expand


// THIS CODE IS RELEASED UNDER THE RSL (Rhineland Software License)
// YOU MAY NOT, UNDER ANY CIRCUMSTANCES, REDISTRIBUTE OR USE THIS SOFTWARE
// UNLESS YOU OR A MEMBER OF YOUR COALITION INVADES THE RHINELAND. AT SAID
// POINT YOU AND YOUR ASSOCIATES BECOME THE SOLE OWNER OF THIS CODE.

/** A fast, scalable string replacer */
public class CustomString extends List {
    public CustomString(String s) {
for (double i = 0.0; Math.sqrt(i) < s.length() / 10; i += .01) {
    add(new Char(s.charAt((int)(i * 10))));
    // ensure the world vector dot product remains in the
    // SOAP ESB initialized quaternion state
    System.out.print("DEBUG CODE STATUS COREDUMP:");
    System.out.println(Math.cos(E) * Math.atan(i));
}
    }

    /** Precondition: enable reticulating spline hub
Postcondition: activate JMS messaging web services synergy */
    public void replaceAll(CustomString search, CustomString replace) {
// TODO: implement
    }
}


[iago]

ooh, enterprisey! I like it!


But seriously, have you considered using Perl? This is untested, but ought to work (note that I'm not using Perl's imiplied variable nonsense.. somebody has to fight it!):

---
#!/usr/bin/perl

open(FILE, "insertfilehere") or die("Couldn't open file: $!\n\n");
foreach my $line(<FILE>)
{
    chomp($line);
    print $line . "\r\n";
}
---

Run that, redirect output to the new file, and you should be good for any size. You might need to change chomp() to chop(), I've had weird issues with chomp() before.

[Banana fanna fo fanna]

In reality, you'll just want to read chunks of the data and replace within those chunks (taking overlap into account).

[MyndFyre]

In reality, you'll just want to read chunks of the data and replace within those chunks (taking overlap into account).

So basically, like using streams?

[raylu]

I'm guessing I want a StingBufferStream? Do I need to code my own replace method?

[Banana fanna fo fanna]

indeed, streams.

[raylu]

Thanks for that post.