x


Why is reading some text files with JS so darn slow?! (title changed to reflect answers)

Here's the relevant code:

var textAsset : TextAsset;

var lineCounter = 0;

function Start() {

    readHugeFile();
}

function readHugeFile() {


    var startTimer = Time.realtimeSinceStartup;

    if (textAsset == null) return;
    reader = new StringReader(textAsset.text);
    line = reader.ReadLine();
    while (line != null)
    {
        lineCounter++;
        line = reader.ReadLine();
    }

    var timeExpired = Time.realtimeSinceStartup - startTimer;

    print("Time Expired:");
    print(timeExpired);

    print("Number of Lines:");
    print(lineCounter);

}

Ok, by points.

  1. I'm not interested in alternatives to reading these files. Alternative/smarter code would be super welcome!

  2. This code is to darn SLOW when the number of lines grows.

  3. This thing scales weirdly. The length of time taken isn't linear with the number of lines in the file.

Some random sampling of text files of different sizes.

                                          (Edits after modification suggested)

Number of Lines   Lines Read / second  |  *Lines Read / second

57,012            251                     518,000
54,818            245                     550,000
41,239            359                     412,000
38,138            385                     545,000
32,305            447                     538,000
25,908            551                     518,000
21,098            681                     527,000
17,668            803                     589,000
16,038            891                     535,000
13,118           1161                     656,000
11,616           1263                     581,000
2,293            6551                     573,000

WTH?

Thanks for any answers, pointers, or directions for better code. Am I doing something silly here?

*Notes The large variance in the number in the far right column are a result of variances in actual line lengths in the files, rounding error, and poor use of significant digits by myself.

With the largest file tested here, the suggested change of code represents a 2000X speed increase!

more ▼

asked Feb 28, 2011 at 04:48 PM

rule62 gravatar image

rule62
79 45 36 41

Never worked with StringReader. How long does it take to execute just the line new StringReader(textAsset.text);? I guess the StringReader doesn't work very efficient on the input data internally. Looks really strange.

Feb 28, 2011 at 05:43 PM Bunny83
Feb 28, 2011 at 05:45 PM Bunny83

Thanks Bunny! Indeed, your link explains the strange scaling. Super helpful. I hate just going "wth? I don't get it."

Feb 28, 2011 at 09:23 PM rule62
(comments are locked)
10|3000 characters needed characters left

1 answer: sort voted first

How about

if (textAsset == null) return;
var lines = textAsset.text.Split("\n"[0]);
var lineCounter = lines.Length;

Might have to be "\r" if that's how lines are delineated.

more ▼

answered Feb 28, 2011 at 05:32 PM

Eric5h5 gravatar image

Eric5h5
107k 51 180 648

Well "Slap my hind with a melon rind"!

Check this out! Editing table in post with new data.

Feb 28, 2011 at 09:25 PM rule62

Eric, this seems like a whopper of a small easily-made error. Should this be brought to anyone's attention? This teensy code-change just saved a project that I was about to give up on.

Feb 28, 2011 at 09:36 PM rule62

@rule62: Well, it depends on what you're doing. Using StringReader with ReadLine may be correct for certain cases. I wasn't sure if my suggestion would be appropriate for what you're doing (since I don't know what that actually is), which is why I phrased my answer as a question. :) Glad it worked for you though.

Mar 01, 2011 at 01:09 AM Eric5h5

After reading Bunny83's link, it sounds like a bug report would be in order. Seems like that should have been fixed given the version of Mono Unity uses?

Mar 01, 2011 at 01:14 AM Eric5h5
(comments are locked)
10|3000 characters needed characters left
Your answer
toggle preview:

Up to 2 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Topics:

x839
x839
x341
x228
x110

asked: Feb 28, 2011 at 04:48 PM

Seen: 1613 times

Last Updated: Feb 28, 2011 at 09:34 PM