Page 1 of 1

How do you open a very large text file? (more than 1GB)

Posted: Thu Aug 20, 2015 2:18 pm
by Outsourced
Hello,

my computer is struggling to open a text file.

My internal fan isn't working either so if I try wordpad, it gets too hot and it shuts down by itself. :(

Re: How do you open a very large text file? (more than 1GB)

Posted: Thu Aug 20, 2015 2:30 pm
by dunbarx
Hi.

I cannot think this is a LiveCode issue.

What happens if you try to do anything else with a large file? Like simply copy and paste it?

Craig Newman

Re: How do you open a very large text file? (more than 1GB)

Posted: Thu Aug 20, 2015 2:45 pm
by Outsourced
dunbarx wrote:Hi.

I cannot think this is a LiveCode issue.

What happens if you try to do anything else with a large file? Like simply copy and paste it?

Craig Newman

sorry :(

Nothing wrong happens. The problem is only opening it.

Re: How do you open a very large text file? (more than 1GB)

Posted: Thu Aug 20, 2015 3:16 pm
by MaxV
1Gb?
What is it?
Is just a plain text file?
I this case you can speed up very easy.
If you use Linux I suggest to split the file:

Code: Select all

split -b 100m  myfile.txt 
so you'll have just 10 files of 100 Mb each.

I think that also Mac has the split command (see https://developer.apple.com/library/mac ... lit.1.html )

Re: How do you open a very large text file? (more than 1GB)

Posted: Thu Aug 20, 2015 4:01 pm
by Outsourced
MaxV wrote:1Gb?
What is it?
Is just a plain text file?
I this case you can speed up very easy.
If you use Linux I suggest to split the file:

Code: Select all

split -b 100m  myfile.txt 
so you'll have just 10 files of 100 Mb each.
yes just a huge text file, unfortunately I only have Windows on this machine.

Thanks

Re: How do you open a very large text file? (more than 1GB)

Posted: Thu Aug 20, 2015 4:20 pm
by FourthWorld
I recently had occasion to parse contents out of a 6.2 GB file (tip: don't try anything bigger than 2 GB in versions prior to 7.1rc1, as seek addressing has been enhanced for 64-bit file systems). So with the most recent version it's possible to handle just about any size file any modern file system allows. The trick is to do so as other programs do: don't attempt to put all that data into RAM at once, but instead page it in chunks and parse what you need as you go.

The specific of how to do that will vary depending on the task at hand. May I ask what you want to do with the data? That will help guide us toward finding a good solution for you.

Re: How do you open a very large text file? (more than 1GB)

Posted: Thu Aug 20, 2015 5:00 pm
by Outsourced
FourthWorld wrote:I recently had occasion to parse contents out of a 6.2 GB file (tip: don't try anything bigger than 2 GB in versions prior to 7.1rc1, as seek addressing has been enhanced for 64-bit file systems). So with the most recent version it's possible to handle just about any size file any modern file system allows. The trick is to do so as other programs do: don't attempt to put all that data into RAM at once, but instead page it in chunks and parse what you need as you go.

The specific of how to do that will vary depending on the task at hand. May I ask what you want to do with the data? That will help guide us toward finding a good solution for you.

I need to extract the email addresses from it.

Searching on Google I just found this :

edit: the forum says "Your account does not have permission to post links or domain/page references."

so I will split the link without http:


codeproject (dot) com /Articles/4583/Extracting-EMail-Addresses-From-a-Document-or-Stri

But have no idea how to run the script. I am a total noob. Thank you

Re: How do you open a very large text file? (more than 1GB)

Posted: Fri Aug 21, 2015 12:27 pm
by MaxV