Page 1 of 1
FTP - force ASCII instead of Binary
Posted: Wed Dec 28, 2016 1:57 am
by robl
Ran into a situation here that I haven't figured out how to work around the issue. I am using FTP to talk to an old IBM Mainframe to retrieve some raw data. The problem is that if I use the following code
Code: Select all
put url "ftp://user:password@servername.server.com/SOME.DATA.REPORTS/REPORT1" into fld "output"
it retrieves the data as binary, which ends being raw EBCDIC (not ASCII) text. If I use the old command line ftp client or a GUI FTP client like FileZilla, I can switch it to ASCII and then transfer the file, resulting in the correct format. "ascii" and "binary" are standard command/modes for the FTP protocol, but LiveCode seems to only do binary?
With the Linux/Unix FTP CLI client, this works as follows
Code: Select all
ftp X.X.X.X
Connected to X.X.X.X.
220 *** MVS FTP Daemon on X.X.X.X ***
Name (X.X.X.X:USER):
331 Okay, waiting for password.
Password:
230 You are now logged in.
Remote system type is MVS.
ftp> ascii
200 Type set to A
ftp> cd SOME.DATA.REPORTS
250 CWD command successful
ftp> get REPORT1
local: REPORT1 remote: REPORT1
227 Entering Passive Mode (X,X,X,X,194,140)
150 Now opening data connection
18615 1.69 MiB/s
226 Transfer complete!
18615 bytes received in 00:00 (1.67 MiB/s)
ftp> quit
221 Bye!
I have yet to find a way to tell LiveCode to grab the file as ASCII. Any suggestions?
Re: FTP - force ASCII instead of Binary
Posted: Wed Dec 28, 2016 7:51 am
by shaosean
I think that the default transfer mode for FTP is binary and seeing as there is no file extension, the server (and perhaps LC) do not know to switch to ASCII mode.. If you want more control over the transfer method, you might want to look at the other FTP library which allows you to switch modes as you see fit..
Re: FTP - force ASCII instead of Binary
Posted: Wed Dec 28, 2016 9:22 am
by strongbow
Can you use
libURLftpCommand
to set it to ASCII before getting the file?
Re: FTP - force ASCII instead of Binary
Posted: Wed Dec 28, 2016 4:49 pm
by jacque
Another likely possibility is that the files are unicode and need to be run through textDecode().
Re: FTP - force ASCII instead of Binary
Posted: Wed Dec 28, 2016 5:29 pm
by FourthWorld
Interesting problem. I've never worked with EBCDIC before, so I checked out the Wikipedia page on it, which includes a section titled "Criticism and Humor":
https://en.wikipedia.org/wiki/EBCDIC#Cr ... _and_humor
It's such an odd and old format that I'm wondering: when you use text mode in other software, where is the translation from EBCDIC to ASCII taking place? Does the FTP client include this, or is there something in the FTP server that handles that?
If the latter then using the suggestions here to request text mode may suffice. If the former, you may need to write a translation function for it. There may already be an EBCDICtoASCII function somewhere in our community, but it's such an old format and so seldom used it seems at least as likely that there may not be one yet.
Re: FTP - force ASCII instead of Binary
Posted: Wed Dec 28, 2016 6:01 pm
by jacque
I missed the ebcdic reference. There's a conversion table here:
http://www.simotime.com/asc2ebc1.htm
Re: FTP - force ASCII instead of Binary
Posted: Wed Dec 28, 2016 8:02 pm
by robl
FourthWorld wrote:Interesting problem. I've never worked with EBCDIC before, so I checked out the Wikipedia page on it, which includes a section titled "Criticism and Humor":
https://en.wikipedia.org/wiki/EBCDIC#Cr ... _and_humor
It's such an odd and old format that I'm wondering: when you use text mode in other software, where is the translation from EBCDIC to ASCII taking place? Does the FTP client include this, or is there something in the FTP server that handles that?
If the latter then using the suggestions here to request text mode may suffice. If the former, you may need to write a translation function for it. There may already be an EBCDICtoASCII function somewhere in our community, but it's such an old format and so seldom used it seems at least as likely that there may not be one yet.
It's actually the server that converts it to ascii when requested to do so. The client sends a "TYPE A" and the server should translate it to ASCII, which happens with using ncftp, "ftp" (CLI), and FileZilla GUI.
Re: FTP - force ASCII instead of Binary
Posted: Wed Dec 28, 2016 8:10 pm
by robl
strongbow wrote:Can you use
libURLftpCommand
to set it to ASCII before getting the file?
Thank you! It almost works. I'm not sure if it is something I am missing, but I can get to download ASCII using your solution, but I have to download it TWICE! If I send a "QUIT" to the ftp server to formally disconnect, and then try, the first time I download, I get a binary file. I download the SAME file, I get ASCII. Very odd!
The code below works -- but notice the 2 downloads of the same file. the first one returns binary (raw EBCDIC) and the second will return ASCII.
Code: Select all
on mouseUp
put empty into fld "output"
downloadTEST
end mouseUp
on downloadTEST
put libURLftpCommand("TYPE A",lServer) & return into fld "status" -- tells the server to send ASCII for ALL downloads on this connection until I tell otherwise
libURLDownloadToFile "ftp://" & lServer & lServerPath & lServerFile, lLocalFile,"downloadFile" -- (I get EBCDIC)
end downloadTEST
on downloadFile
libURLDownloadToFile "ftp://" & lServer & lServerPath & lServerFile, lLocalFile,"downloadDone" -- (this is ASCII)
end downloadFile
on downloadDone
put URL ("file:"& lLocalFile) after fld "output"
killFTP
end downloadDone
on killFTP
put libURLftpCommand("QUIT",lServer) & return into fld "status"
end killFTP
Re: FTP - force ASCII instead of Binary
Posted: Thu Dec 29, 2016 12:26 am
by strongbow
Did you try a small "wait" between setting to Ascii and getting the file?
Re: FTP - force ASCII instead of Binary
Posted: Thu Dec 29, 2016 3:13 am
by robl
strongbow wrote:Did you try a small "wait" between setting to Ascii and getting the file?
Indeed, I tried a delay of a varying amounts up to 10 seconds, still the same results. I'm almost wondering if I am missing something in the "raw" FTP commands that isn't fully initializing it correctly and the libURLdownloadToFile finishes the initialization as part of the first download.
If I download 10 files, the first is binary, the remaining 9 are ASCII. Something is happening differently during the first download attempt and I haven't quite figured it out yet. I may end up trying wireshark and comparing what livecode is sending to that of another client.
Re: FTP - force ASCII instead of Binary
Posted: Thu Dec 29, 2016 1:24 pm
by robl
I think I found a solution, and it is very simple, but not documented in the LiveCode docs.
Code: Select all
libURLDownloadToFile "ftp://" & lServer & lServerPath & lServerFile & ";TYPE=A", lLocalFile,"downloadDone"
That's it. No need to use any libURLftpCommand. No need to download the file twice.
Just add the string ";TYPE=A" to the end of the "ftp://" URL. i.e., "
ftp://my.server.com/some/path/file;TYPE=A". It only seems to work with libURLDownloadToFile, and not the "put URL" command.
I found the solution while looking at the documentation for the command line "curl" tool.
https://curl.haxx.se/docs/manpage.html
-B, --use-ascii
(FTP LDAP) Enable ASCII transfer. For FTP, this can also be enforced by using an URL that ends with ";type=A". This option causes data sent to stdout to be in text mode for win32 systems.
Re: FTP - force ASCII instead of Binary
Posted: Thu Dec 29, 2016 11:54 pm
by strongbow
Ahh, that's right! I'd forgotten that! Probably last time I used that form was in the 80s! Great that you found it tho'.