Get the last two columns

[richmond62]

To extract the last two items from each line into a new line in a new variable took 46 seconds

Damn: wouldn't even have time for a gulp of coffee.

Using WHICH script?

[dunbarx]

Richmond.

Using WHICH script?

Why, ANY handler that does the job, no?

Code: Select all

on mouseUp
   put "A" & tab & "B" & tab & "C" & tab & "D" & tab & "F" & return into temp
   repeat 100000000
      put temp after dataSet
   end repeat

   set the itemDel to tab
   put the ticks into aa  --start timing
   repeat for each line tLine in dataSet
      put item -2 to -1 of tLine & return after accum
   end repeat
   answer the ticks - aa
end mouseUp

Craig

[xoxiwe]

hello guys, hope you're doing well.
I'm sry for the late reply as my pc was giving troublesome, so I had to send it to servicing center. Anyway, let's come to the topic.

First I've tried the below code written by me with the help of richmond62:

Code: Select all

open file dFile for write -- dFile is the destination text file LC will write the desired output
open file sFile for read -- sFile is the source text file that holds the raw data
set the itemdel to ","
put 1 into tCount
repeat until line tCount of sFile is empty
   put line tCount of sFile into tLine
   write item -2 to -1 of tLine & cr to file dFile
   add 1 to tCount
end repeat
close file dFile
close file sFile

It just went crap since it was writing the source file name with it's full path in a repeat loop. Just threw it to the trash as I realized I definitely did something wrong in the code. Actually I didn't take the source data into variable because the data is huge, and my pc wont be able to handle it taking the data into memory; so I think I must have made some mistake for not letting LC to read the data of the source file in a proper way.

Then I tried Bob's code, and it went well until the last line of the source data, then it started making unlimited lines with just ",". So to end up the repeat loop, I just had to add if it is not empty then condition after the line read from file myFilePath for 1 line, and the repeat stops after processing the last line, and I get the desired result as expected. Thanks Bob, your code was really worth trying

Thanks to all who spent their valuable time to help me solve the problem

[bogs]

Bogs, I created a list of 100 million lines, each line containing five chars separated by tabs, about one GB.
To extract the last two items from each line into a new line in a new variable took 46 seconds.
This on an M2 Mac Mini.

Craig

I don't know that an m2 mac mini qualifies as a 'severely limited resource pc' such as the OP states having. From the OP:

my pc might not be able to handle 300-500 mb size data

Being unfamiliar with Mac offerings say, newer than 1998, I went and looked up the mini's specs.

Code: Select all

M2 mac mini specs - 
	8-core CPU <- not qualified as 'limited resources'>
	10-core GPU <- not qualified as 'limited resources'>
	16-core Neural Engine <- not qualified as 'limited resources'> 
	options, 8GB or 16GB of unified memory <- not qualified as 'limited resources'>
	storage options, 256GB to 512GB SSD <- not qualified as 'limited resources'>

If the OP had your banging mini, I'm pretty sure they could suck up a 4GB file in memory with no issues what so ever

Even with all that horsepower, it took 46 seconds for your machine to do that one GB. Slam that same 100 million lines into an array, divide it, and shove the last 2 keys into 2 separate fields, tell me how it compares? I'll be willing to lay a large fiver on which I suspect goes faster

[bogs]

Bogs.

I created a list of 100 million lines, each line containing five chars separated by tabs, about one GB.

To extract the last two items from each line into a new line in a new variable took 46 seconds.

This on an M2 Mac Mini.

Craig

I suspect a lot of the delay in the testing bogs did might have been from the updates made in the message box. That would certainly create an overhead in field display updates.
The wait with messages inside the loop is a valuable way to stop the ui locking up though.

Surprisingly, not so much. The OP stated they had a limited pc, one that would struggle with bringing 300-500 MB files in. While I private messaged them to ask for more detailed specs, I had not received an answer to that request as of this post.

The testing I was doing therefore was in a virtual machine with 2 cores and 4 GB of ram, approximating a system I had some 25 years ago. If their machine is more limited than that, which is possible, then the times I listed would probably be quadrupled.

Further, the results I posted were without the message box, I used that because some routines mentioned in this thread ran SO slowly, I wanted to make sure that the process ran all the way through. During testing on 2 of the routines mentioned, the process locked up, and one waited all the way till the last line of the file to do it.

Lastly, with a machine so limited that 500 MB in ram would crush it, at some point before that limit you would have to put the variable into something to hold it/off load it at some point and flush the variable would be my thought. File operations make field ops look like they are lightning fast by comparison.

Odd side note, even though I limited that VM to 4 GB ram, when I eventually tested with a 1 GB file, it did not proportionately use 1 GB more ram. I am unsure why that would be.