This discussion is archived
3 Replies Latest reply: Jul 10, 2012 8:28 AM by slsam01 RSS

How to avoid Unix stripping NUL character

slsam01 Newbie
Currently Being Moderated
Hello,

I have a text file that contains NUL character.

If I open the file in Windows, I can see the NUL character rendered in a certain way.

If I open the file in vi or cat, I don't see the NUL character rendered.

If I use sed or tr to replace the NUL character with ?, The NUL characters are simply disappeared rather than replaced.
For example: sed 's/\x0/?/g' input.txt > output.txt

Even if I opened the file in vi in Unix and saved it without change, the NUL character also disappeared.

It seems to me that Unix simply strip NUL character.

Could you please advise how Unix handle NUL character? How can I replace the NUL character with another character (for example ?) , rather than just simply disappeared?

My Unix OS is: SunOS 5.10 Generic_142909-17 sun4v sparc SUNW,SPARC-Enterprise-T5220

Many thanks,
  • 1. Re: How to avoid Unix stripping NUL character
    user171873 Explorer
    Currently Being Moderated
    Historically, Unix has not treated the null character (zero) as a valid text character. The C programming language uses it to indicate the end of a string. Consequently, when a C program (such as sed or vi) loads text which contains embedded nulls from a file, nulls need to be stripped to prevent them from prematurely terminating the input string. Nulls are also sometimes used for padding. So, yes the null's tend to get stripped by the text processing commands.

    The nulls in the files are maintained by programs which don't treat the contents of the file as text. For example, the command "od -c <file>" should print out the nulls in a file as "\0" which will allow you to determine if they are still there. If this were my problem, I'd probably whip up a little program to read the file (as a binary file) and substitute another character (which won't get stripped) for the nulls. I'm afraid I can't think of an existing program which would do this offhand. Perhaps a binary editor could be used? I found:

    http://www.sunmanagers.org/pipermail/summaries/2003-January/003026.html

    but I have no experience with the programs suggested here. I have no idea whether they would work for you. Perhaps someone else has a better suggestion.
  • 2. Re: How to avoid Unix stripping NUL character
    Marty Newbie
    Currently Being Moderated
    I bump into that often enough that years ago (decades ago?) I slapped together the following C program. Put it into a file such as null2spc.c with your favorite text editor then run the command "make" to compile it. Redirect in and out, e.g.

    null2spc < input_file_with_nulls > output_file_without_nulls

    Good luck. Code follows.

    #include <stdlib.h>
    #include <stdio.h>

    // turn nulls into spaces

    // NO ERROR CHECKING -- AT ALL!!!!

    int main(void)
    {
    int c; // generic char var
    // file pointers
    FILE* fpi=stdin;
    FILE* fpo=stdout;

    while ((c=fgetc(fpi))!=EOF) {
    // fix up null
    if (c==0)
    fputc(' ', fpo);
    // pass through all else
    else
    fputc(c, fpo);
    }

    return 0;
    }
  • 3. Re: How to avoid Unix stripping NUL character
    slsam01 Newbie
    Currently Being Moderated
    Thank you for your information.

    Edited by: slsam01 on Jul 10, 2012 8:28 AM

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points