x


Why is reading some text files with JS so darn slow?! (title changed to reflect answers)

Here's the relevant code:

var textAsset : TextAsset;

var lineCounter = 0;

function Start() {

 readHugeFile();

}

function readHugeFile() {

 var startTimer = Time.realtimeSinceStartup;

 if (textAsset == null) return;
 reader = new StringReader(textAsset.text);
 line = reader.ReadLine();
 while (line != null)
 {
     lineCounter++;
     line = reader.ReadLine();
 }

 var timeExpired = Time.realtimeSinceStartup - startTimer;

 print("Time Expired:");
 print(timeExpired);

 print("Number of Lines:");
 print(lineCounter);

}

Ok, by points.

  1. I'm not interested in alternatives to reading these files. Alternative/smarter code would be super welcome!

  2. This code is to darn SLOW when the number of lines grows.

  3. This thing scales weirdly. The length of time taken isn't linear with the number of lines in the file.

Some random sampling of text files of different sizes.

(Edits after modification suggested)

Number of Lines Lines Read / second | *Lines Read / second

57,012 251 518,000 54,818 245 550,000 41,239 359 412,000 38,138 385 545,000 32,305 447 538,000 25,908 551 518,000 21,098 681 527,000 17,668 803 589,000 16,038 891 535,000 13,118 1161 656,000 11,616 1263 581,000 2,293 6551 573,000

WTH?

Thanks for any answers, pointers, or directions for better code. Am I doing something silly here?

*Notes The large variance in the number in the far right column are a result of variances in actual line lengths in the files, rounding error, and poor use of significant digits by myself.

With the largest file tested here, the suggested change of code represents a 2000X speed increase!

more ▼

asked Feb 28, 2011 at 04:48 PM

rule62 gravatar image

rule62
79 45 37 44

Never worked with StringReader. How long does it take to execute just the line new StringReader(textAsset.text);? I guess the StringReader doesn't work very efficient on the input data internally. Looks really strange.

Feb 28, 2011 at 05:43 PM Bunny83
Feb 28, 2011 at 05:45 PM Bunny83

Thanks Bunny! Indeed, your link explains the strange scaling. Super helpful. I hate just going "wth? I don't get it."

Feb 28, 2011 at 09:23 PM rule62
(comments are locked)
10|3000 characters needed characters left

1 answer: sort voted first

How about

if (textAsset == null) return;
var lines = textAsset.text.Split("n"[0]);
var lineCounter = lines.Length;

Might have to be "r" if that's how lines are delineated.

more ▼

answered Feb 28, 2011 at 05:32 PM

Eric5h5 gravatar image

Eric5h5
112k 56 203 694

Well "Slap my hind with a melon rind"!

Check this out! Editing table in post with new data.

Feb 28, 2011 at 09:25 PM rule62

Eric, this seems like a whopper of a small easily-made error. Should this be brought to anyone's attention? This teensy code-change just saved a project that I was about to give up on.

Feb 28, 2011 at 09:36 PM rule62

@rule62: Well, it depends on what you're doing. Using StringReader with ReadLine may be correct for certain cases. I wasn't sure if my suggestion would be appropriate for what you're doing (since I don't know what that actually is), which is why I phrased my answer as a question. :) Glad it worked for you though.

Mar 01, 2011 at 01:09 AM Eric5h5

After reading Bunny83's link, it sounds like a bug report would be in order. Seems like that should have been fixed given the version of Mono Unity uses?

Mar 01, 2011 at 01:14 AM Eric5h5
(comments are locked)
10|3000 characters needed characters left
Your answer
toggle preview:

Up to 2 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Topics:

x934
x915
x366
x247
x122

asked: Feb 28, 2011 at 04:48 PM

Seen: 1743 times

Last Updated: Feb 28, 2011 at 09:34 PM