Help - How 2 open and read large files over 32,768 bytes?

[Android]

Can anyone advise how to open and read a file larger than 32.7k that contains multiple records with one cariage return at the end of the file? The file is field delimited with tilde (~) and asterisks (*) that are showing up as "square boxes" in the sample. Here's the sample of the data...

ISA00 00 1234567 AA8901234 5678901234A004012345678900P:GSMD123456789012200603241234567X004010X091A1... etc.

I realize the 32768 size limit is an inherant QB limitation (it bombs out after reaching 32768 characters), but I'm hoping to figure out a way to have QuickBasic do it so I don't have to shell out to some other program (that I haven't yet found) to do it.

Any help will be greatly appreciated!

Android

[moneo]

You mention a file grerater than 32K, but I assume that the problem is that you have records that are over 32K each. If this is true, you have a real problem.

Even if you opened the file for binary and read the records in smaller chunks, you still may have problems if you need to have an entire record in memory in order to scan it and process it.

You mention delimiters. Is it possible that each physical record has delimiters within it to separate the logical records that need to be processed individually? In other words, maybe you don't need to process an entire record all at once, only the delimited pieces one at a time.

By the way, your sample doesn't shed any light on the problem.

Where did you get this file from? Would it be possible to get the file with smaller records?
*****

[{Nathan}]

You could try implimenting it into a QB file as Data and '$INCLUDE: ing it,,, but then you wouldn't be able to have it all in one time...

*cough* *freebasic for dos* *cough*

[Android]

Here is the actual code we're using. The problem starts when the ZX is greater than 32767 and we try to do the FIELD statement with the ZX .

OPEN "R",1,file2$,1
FIELD 1,1 AS MEF$
ZX=LOF(1) ' this gets the file size which is also the record size since it contains only 1 record
CLOSE 1

OPEN "R",#1,file2$,zx ' this open with ZX which is the size

FIELD #1,ZX AS MEF$ ' this reads with ZX as the amount to read. If ZX is larger than 32767 it bombs
GET #1, 1

I can't post a sample file because the delimiters don't show up correctly. However, the Delimiters are specified in the first header segment, a 105 byte fixed length record.

The data element separator is byte number 4; the component element separator is byte number 105; and the segment terminator is the byte that immediately follows the component element separator.

The delimiters are:
* Asterisk Data Element Separator
: Colon Subelement Separator
~ Tilde Segment Terminator

I hope this helps to understand the problem and (maybe) someone can come up with a solution!

Android

[MystikShadows]

Maybe this could solve your problem:

Code: Select all

'$DYNAMIC
DIM WorkString() AS STRING
DIM Counter AS INTEGER
DIM LineCounter AS INTEGER
DIM Character AS STRING * 1

OPEN file2$ FOR BINARY AS #1
Counter = 0
REDIM WorkString(1 to 10) AS STRING
LineCounter = 1
DO WHILE NOT EOF (#1) OR 
    Counter = Counter + 1 
    GET #1, , Character    
    IF Counter  > 32767 THEN
        LineCounter = LineCounter + 1
        Counter = 1
    End If
    WorkString(LineCounter) = WorkString(LineCounter) + character
LOOP

This should effectively get the contents of a "record" bigger than 32768 by splitting the characters read into lines of 32767 characters each. Your reading character by characters, performances just might go down , but it will do the job.

However, from what I'm reading here, it seems you don't need to read everything at the same time. It seems you only need to read between the :, * and ~ and do whatever with them. If so, it's easy to do that and you don't need a bigger variable.

Just let me know what happens with text between the :, * and ~ and I'm sure we can come up with a solution.

Hope this helps

[Android]

Hey there MystikShadows! Thanks for your idea. This looks like it might work but I'm not clear on some of it and hesitate using code I don't understand (in case it needs altering for some reason later on). Could I trouble you to explain to me what each line of code is doing here?

Thanks in advance,
Android

[MystikShadows]

Of course you can here goes:

Code: Select all

' This declares the WorkString dynamic array.  '$DYNAMIC is needed
' to accomplish this.
'$DYNAMIC                               
DIM WorkString() AS STRING      
DIM Counter AS INTEGER     ' Character counter current length read
DIM LineCounter AS INTEGER ' Will count lines needed for reading.
DIM Character AS STRING * 1' our one character buffer to read from file

' This opens the data file as a binary file (where the data type
' of the variable used determines the number of bytes read 
' in a GET statement.
OPEN file2$ FOR BINARY AS #1

' We start the counter at 0.  It will increment by one everytime
' we read a character from the file.
Counter = 0

' Change this redim statement if you need more lines to work with
REDIM WorkString(1 to 10) AS STRING
' LineCounter starts as 1
LineCounter = 1

DO WHILE NOT EOF (#1) 
    Counter = Counter + 1              ' Increment Counter by one
    GET #1, , Character                ' Read 1 character from the file
    IF Counter  > 32767 THEN           ' If Counter is greater than 32767
                                       ' it's time to change line number 
                                       ' and reset counter to 1     
        LineCounter = LineCounter + 1
        Counter = 1
    End If
    ' We add the character read to the WorkString array we declared
    ' at LineNumber.
    WorkString(LineCounter) = WorkString(LineCounter) + Character
LOOP

Basically, I read one character at a time from the file and add that character to the array of string WorkString(). Best way to explain it is thing of the workstring array as a matrix of 32767 columns by X number of lines. but i just used a string variable which can be 32767 characters long instead of actually creating a string matrix.

If I read 32767 characters, i've reached the limit of the STRING data type
so I position myself at the 2nd element of the array and set Counter back to 1 so I'm readt to start reading into a new line fo string until Counter reaches 32767 again or I reach the end of file.

Let me know if I can clarify this further for you.

[moneo]

Mystic, Just out of curiosity, what does "OR" mean on the following statement that you provided?

DO WHILE NOT EOF (#1) OR

*****

[MystikShadows]

I corrected it, I'm not even sure where that OR came from. might have pasted something by typing to fast .

Thanks for pointing it out...