3 Replies Latest reply: Jul 10, 2012 10:28 AM by slsam01 RSS

    How to avoid Unix stripping NUL character


      I have a text file that contains NUL character.

      If I open the file in Windows, I can see the NUL character rendered in a certain way.

      If I open the file in vi or cat, I don't see the NUL character rendered.

      If I use sed or tr to replace the NUL character with ?, The NUL characters are simply disappeared rather than replaced.
      For example: sed 's/\x0/?/g' input.txt > output.txt

      Even if I opened the file in vi in Unix and saved it without change, the NUL character also disappeared.

      It seems to me that Unix simply strip NUL character.

      Could you please advise how Unix handle NUL character? How can I replace the NUL character with another character (for example ?) , rather than just simply disappeared?

      My Unix OS is: SunOS 5.10 Generic_142909-17 sun4v sparc SUNW,SPARC-Enterprise-T5220

      Many thanks,
        • 1. Re: How to avoid Unix stripping NUL character
          Historically, Unix has not treated the null character (zero) as a valid text character. The C programming language uses it to indicate the end of a string. Consequently, when a C program (such as sed or vi) loads text which contains embedded nulls from a file, nulls need to be stripped to prevent them from prematurely terminating the input string. Nulls are also sometimes used for padding. So, yes the null's tend to get stripped by the text processing commands.

          The nulls in the files are maintained by programs which don't treat the contents of the file as text. For example, the command "od -c <file>" should print out the nulls in a file as "\0" which will allow you to determine if they are still there. If this were my problem, I'd probably whip up a little program to read the file (as a binary file) and substitute another character (which won't get stripped) for the nulls. I'm afraid I can't think of an existing program which would do this offhand. Perhaps a binary editor could be used? I found:


          but I have no experience with the programs suggested here. I have no idea whether they would work for you. Perhaps someone else has a better suggestion.
          • 2. Re: How to avoid Unix stripping NUL character
            I bump into that often enough that years ago (decades ago?) I slapped together the following C program. Put it into a file such as null2spc.c with your favorite text editor then run the command "make" to compile it. Redirect in and out, e.g.

            null2spc < input_file_with_nulls > output_file_without_nulls

            Good luck. Code follows.

            #include <stdlib.h>
            #include <stdio.h>

            // turn nulls into spaces

            // NO ERROR CHECKING -- AT ALL!!!!

            int main(void)
            int c; // generic char var
            // file pointers
            FILE* fpi=stdin;
            FILE* fpo=stdout;

            while ((c=fgetc(fpi))!=EOF) {
            // fix up null
            if (c==0)
            fputc(' ', fpo);
            // pass through all else
            fputc(c, fpo);

            return 0;
            • 3. Re: How to avoid Unix stripping NUL character
              Thank you for your information.

              Edited by: slsam01 on Jul 10, 2012 8:28 AM