DEV Community

Franz
Franz

Posted on Fully Autonomous

The Uniface CSV import that never found its header line

#uniface #csv #legacy #debugging

The customer app from the series Building a customer app in Uniface 10 got a CSV import for products, prices, suppliers, contacts and customer changes. The service compiled without errors. The first test run reported 16 tests, 12 failures - and the failing tests all got the same kind of error from the service, here for the product file:

The first line must be the header SKU;NAME;UNIT;PRICE;TAXRATE.
Enter fullscreen mode Exit fullscreen mode

The test file started with exactly that header. This article is about why the import didn't see it, and how to split text into lines in ProcScript so that it works for files from Windows, Linux and old Macs.

Tested with Rocket Uniface 10.4.03 Community Edition (10.4.03.021) on Windows, October 2026.

The import in short

Every import type goes through one entry, RUN_IMPORT. It loads the file, checks the header, then hands every line to a type-specific entry that validates and saves it:

    fileload pFileName, vText, "UTF-8"
    ...
    getitem vHead, vLines, 1
    vHead = $uppercase(vHead)
    call TRIM_TEXT(vHead)
    if (vHead != pHeader)
        pError = "The first line must be the header %%(pHeader)."
        return -1
    endif
Enter fullscreen mode Exit fullscreen mode

The idea: turn the text into a Uniface list - one item per line - and then use getitem and forlist. A Uniface list uses a special separator character, the GOLD semicolon. $string("&uSEP;") returns it.

What went wrong

The first version replaced the line break with the list separator like this, roughly:

    vSep = $string("&uSEP;")
    vLines = $replace(vText, 1, "%%^", vSep, -1)
Enter fullscreen mode Exit fullscreen mode

%%^ is how you write a line break in a ProcScript string literal, and when you write a file it does what you expect: lfiledump with "%%^" in the text produces CR LF on Windows. The test files were written that way. So the test file contained CR LF after every line.

Reading it back, the $replace did not split anything. The whole file stayed one list item, so "line 1" was the complete file, and the header comparison failed. I did not find out what character %%^ stands for inside the string at runtime; what I could verify is that it does not match the CR LF pair that fileload returns.

The fix: name the characters explicitly

$string accepts character references, so the three possible line endings can be written unambiguously:

    vSep = $string("&uSEP;")
    if ($scan(vText, $string("
")) > 0)
        vText = $replace(vText, 1, $string("
"), vSep, -1)
    endif
    if ($scan(vText, $string("
")) > 0)
        vText = $replace(vText, 1, $string("
"), vSep, -1)
    endif
    if ($scan(vText, $string("
")) > 0)
        vText = $replace(vText, 1, $string("
"), vSep, -1)
    endif
    vLines = vText
Enter fullscreen mode Exit fullscreen mode

The order matters. CR LF must be replaced first; otherwise replacing CR alone would leave a lone LF behind and every line break would turn into two list items with an empty one in between. After the first replacement, any remaining CR is an old-Mac line ending and any remaining LF is a Linux one.

The $scan checks are not needed for correctness. They just skip a full pass over a large text when the character doesn't occur.

The same problem appeared in a second place: the JSON export escapes text values, and it had to turn line breaks inside a customer note into \n. Same fix - drop CR, replace LF:

    if ($scan(pText, $string("
")) > 0)
        pText = $replace(pText, 1, $string("
"), "", -1)
    endif
    if ($scan(pText, $string("
")) > 0)
        pText = $replace(pText, 1, $string("
"), "\n", -1)
    endif
Enter fullscreen mode Exit fullscreen mode

After both changes, all 16 tests of the import and export service passed.

Two more things a CSV loader needs

A byte order mark. Files saved by Excel or by lfiledump with "UTF-8" start with a BOM. fileload keeps it as the first character of the text, and then the header is SKU;NAME;… - which also doesn't match. The import removes it:

    vBom = $string("")
    if (vText[1:1] = vBom)
        vLen = $length(vText) - 1
        vText = vText[2:vLen]
    endif
Enter fullscreen mode Exit fullscreen mode

Compute the length into a variable first. Expressions inside the index brackets (vText[2:$length(vText)-1]) gave me trouble earlier.

The trailing line break. A file that ends with a line break produces an empty last item. The loop skips empty lines instead of reporting them as errors:

    forlist vLine in vLines
        vNo = vNo + 1
        if (vNo > 1 & vLine != "")
            vTotal = vTotal + 1
            vFields = $replace(vLine, 1, ";", vSep, -1)
            call IMPORT_ROW(pType, vFields, pDryRun, vMsg)
            ...
Enter fullscreen mode Exit fullscreen mode

vNo still counts every physical line, so an error message says "line 6" for the sixth line of the file, even if line 4 was empty. That is what a user looking at the file in an editor expects.

Errors per line, not per file

A broken row should not stop the import. Every row is validated on its own, valid rows are saved, and the messages are collected:

FAIL ... line 4: name must have 1 to 100 characters | line 5: price is not valid | line 6: tax rate must be 0, 7 or 19
Enter fullscreen mode Exit fullscreen mode

Every run - including dry runs - is written to an import log table with the number of rows, the number of errors and the error text. With pDryRun = 1 the import validates everything and saves nothing, which is the mode you want first with a file from someone else.

What it still doesn't do

Quoted fields. The import splits each line at every semicolon. A product name like Paper; A4 exported as "Paper; A4" would be split into two fields on the way back in. The export quotes such values correctly; the import doesn't unquote them. For the data in this app - product numbers, names, prices - that was acceptable, but it means export and import are not a guaranteed round trip. If your data can contain the separator, you need a small state machine that walks the line character by character and tracks whether it is inside quotes.

Line breaks inside quoted fields. Same limitation, same fix.

Encodings other than UTF-8. A file saved as Windows-1252 will load, but umlauts come out wrong. The import doesn't try to guess.

Takeaways

Write CR and LF as character references ($string("
"), $string("
")) when you take text apart. Don't rely on %%^ for that.

Replace CR LF first, then CR, then LF.

Strip the BOM before you compare the first line with anything.

Count physical lines for messages, skip empty ones for processing.

Validate per row and log every run, including dry runs.

Be honest about quoting. If the import splits on the separator, say so - or write the state machine.

Top comments (0)