How do you open a very large text file? (more than 1GB)

Got a LiveCode personal license? Are you a beginner, hobbyist or educator that's new to LiveCode? This forum is the place to go for help getting started. Welcome!

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller

Post Reply
Outsourced
Posts: 4
Joined: Thu Aug 20, 2015 2:15 pm

How do you open a very large text file? (more than 1GB)

Post by Outsourced » Thu Aug 20, 2015 2:18 pm

Hello,

my computer is struggling to open a text file.

My internal fan isn't working either so if I try wordpad, it gets too hot and it shuts down by itself. :(

dunbarx
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 10331
Joined: Wed May 06, 2009 2:28 pm

Re: How do you open a very large text file? (more than 1GB)

Post by dunbarx » Thu Aug 20, 2015 2:30 pm

Hi.

I cannot think this is a LiveCode issue.

What happens if you try to do anything else with a large file? Like simply copy and paste it?

Craig Newman

Outsourced
Posts: 4
Joined: Thu Aug 20, 2015 2:15 pm

Re: How do you open a very large text file? (more than 1GB)

Post by Outsourced » Thu Aug 20, 2015 2:45 pm

dunbarx wrote:Hi.

I cannot think this is a LiveCode issue.

What happens if you try to do anything else with a large file? Like simply copy and paste it?

Craig Newman

sorry :(

Nothing wrong happens. The problem is only opening it.

MaxV
Posts: 1580
Joined: Tue May 28, 2013 2:20 pm
Contact:

Re: How do you open a very large text file? (more than 1GB)

Post by MaxV » Thu Aug 20, 2015 3:16 pm

1Gb?
What is it?
Is just a plain text file?
I this case you can speed up very easy.
If you use Linux I suggest to split the file:

Code: Select all

split -b 100m  myfile.txt 
so you'll have just 10 files of 100 Mb each.

I think that also Mac has the split command (see https://developer.apple.com/library/mac ... lit.1.html )
Livecode Wiki: http://livecode.wikia.com
My blog: https://livecode-blogger.blogspot.com
To post code use this: http://tinyurl.com/ogp6d5w

Outsourced
Posts: 4
Joined: Thu Aug 20, 2015 2:15 pm

Re: How do you open a very large text file? (more than 1GB)

Post by Outsourced » Thu Aug 20, 2015 4:01 pm

MaxV wrote:1Gb?
What is it?
Is just a plain text file?
I this case you can speed up very easy.
If you use Linux I suggest to split the file:

Code: Select all

split -b 100m  myfile.txt 
so you'll have just 10 files of 100 Mb each.
yes just a huge text file, unfortunately I only have Windows on this machine.

Thanks

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 10052
Joined: Sat Apr 08, 2006 7:05 am
Contact:

Re: How do you open a very large text file? (more than 1GB)

Post by FourthWorld » Thu Aug 20, 2015 4:20 pm

I recently had occasion to parse contents out of a 6.2 GB file (tip: don't try anything bigger than 2 GB in versions prior to 7.1rc1, as seek addressing has been enhanced for 64-bit file systems). So with the most recent version it's possible to handle just about any size file any modern file system allows. The trick is to do so as other programs do: don't attempt to put all that data into RAM at once, but instead page it in chunks and parse what you need as you go.

The specific of how to do that will vary depending on the task at hand. May I ask what you want to do with the data? That will help guide us toward finding a good solution for you.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

Outsourced
Posts: 4
Joined: Thu Aug 20, 2015 2:15 pm

Re: How do you open a very large text file? (more than 1GB)

Post by Outsourced » Thu Aug 20, 2015 5:00 pm

FourthWorld wrote:I recently had occasion to parse contents out of a 6.2 GB file (tip: don't try anything bigger than 2 GB in versions prior to 7.1rc1, as seek addressing has been enhanced for 64-bit file systems). So with the most recent version it's possible to handle just about any size file any modern file system allows. The trick is to do so as other programs do: don't attempt to put all that data into RAM at once, but instead page it in chunks and parse what you need as you go.

The specific of how to do that will vary depending on the task at hand. May I ask what you want to do with the data? That will help guide us toward finding a good solution for you.

I need to extract the email addresses from it.

Searching on Google I just found this :

edit: the forum says "Your account does not have permission to post links or domain/page references."

so I will split the link without http:


codeproject (dot) com /Articles/4583/Extracting-EMail-Addresses-From-a-Document-or-Stri

But have no idea how to run the script. I am a total noob. Thank you

MaxV
Posts: 1580
Joined: Tue May 28, 2013 2:20 pm
Contact:

Re: How do you open a very large text file? (more than 1GB)

Post by MaxV » Fri Aug 21, 2015 12:27 pm

Livecode Wiki: http://livecode.wikia.com
My blog: https://livecode-blogger.blogspot.com
To post code use this: http://tinyurl.com/ogp6d5w

Post Reply