This discussion is archived
1 2 Previous Next 22 Replies Latest reply: Dec 21, 2011 6:05 AM by 901561 RSS

[HELP]: Email sent from library running under Tomcat loses French chars

901561 Newbie
Currently Being Moderated
Hi,

I have buill a wrapper jar library which sends an email containing UTF-8 coded chars. Now, when sending an email from a Solaris environment, using the shell directly (using a built-in test method), the mail gets generated correctly (characters are correctly coded in the email body - when checking the log4j - which has been configured to write content in UTF-8) and the email is readable on MS Outlook.

However, when calling the exact same method from a WebApp, the email is generated correctly, however as soon as it is sent, the JavaMail DEBUG output prints the email already with the French chars screwed up.

Here are the file contents and outputs from the log to illustrate what is happening....

I have already taken care of the email generation methods to make sure that all input and output is compliant with UTF-8 encoding (all readers, writers and getBytes() methods explicitly specify the encoding type as "UTF-8" to avoid relying on the JVM default locale setting)...

If someone has suffered from a similar behaviour and/or have some tips on how to solve this, it would be greatly appreciated! :)

Thank you all for your support!
Gizmo


*(1) - The Email in French - Generated internally and retrieved from the Tomcat's log file:*
-----------------------------------------
Cher Utilisateur,

Vous avez commandé auprès d’EUMETSAT les articles suivants:

Description, Quantité, Prix Unitaire

- Meteosat Key Unit(MKU), 1 , EUR700.0
- Meteosat Station Key Unit(SKU), 1 , EUR400.0
- Metop Direct Readout Station Key Unit(SKU), 1 , EUR400.0

Montant Total: EUR1500.0


Nous vous rappelons que vous pouvez maintenant procéder au paiement par carte de crédit via Saferpay. Veuillez cliquer sur le lien ci-dessous et suivre les instructions:

Nous vous ferons parvenir les articles commandés dès réception de votre paiement.

Pour toute question ou si vous souhaitez changer/annuler votre commande, veuillez nous contacter à ops@bla.int.

Cordialement,
-----------------------------------------

*(2) - Output of the JavaMail API with the mail.debug property set to true*
-----------------------------------------
DEBUG: getProvider() returning javax.mail.Provider[TRANSPORT,smtp,com.sun.mail.smtp.SMTPTransport,Sun Microsystems, Inc]
DEBUG SMTP: useEhlo true, useAuth false
DEBUG SMTP: trying to connect to host "mail.bla.int", port 25, isSSL false
220 *********************************************************
DEBUG SMTP: connected to host "mail.bla.int", port: 25

EHLO veopis02_e0
250-listserv.bla.int Pleased to meet you
250-ETRN
250-XXXXXXXA
250-AUTH LOGIN
250-AUTH=LOGIN
250-XXXB
250-XXXC
250-PIPELINING
250-8BITMIME
250-DSN
250 SIZE 10485760
DEBUG SMTP: Found extension "ETRN", arg ""
DEBUG SMTP: Found extension "XXXXXXXA", arg ""
DEBUG SMTP: Found extension "AUTH", arg "LOGIN"
DEBUG SMTP: Found extension "AUTH=LOGIN", arg ""
DEBUG SMTP: Found extension "XXXB", arg ""
DEBUG SMTP: Found extension "XXXC", arg ""
DEBUG SMTP: Found extension "PIPELINING", arg ""
DEBUG SMTP: Found extension "8BITMIME", arg ""
DEBUG SMTP: Found extension "DSN", arg ""
DEBUG SMTP: Found extension "SIZE", arg "10485760"
DEBUG SMTP: use8bit false
MAIL FROM:<noreply@bla.int>
250 Sender OK
RCPT TO:<nuno.viana@bla.int>
250 Recipient OK
DEBUG SMTP: Verified Addresses
DEBUG SMTP: nuno.viana@bla.int
DATA
354 Send message, end with <CRLF>.<CRLF>
From: noreply@bla.int
Reply-To: noreply@bla.int
To: nuno.viana@bla.int
Message-ID: <26508945.0.1321967603800.JavaMail.tomcat@mail.bla.int>
Subject: [EO PORTAL]: Reminder: Credit card payment for your order 10000116
MIME-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Cher Utilisateur,

Vous avez command? aupr?s d?EUMETSAT les articles suivants:

Description, Quantit?, Prix Unitaire

- Meteosat Key Unit(MKU), 1 , EUR700.0
- Meteosat Station Key Unit(SKU), 1 , EUR400.0
- Metop Direct Readout Station Key Unit(SKU), 1 , EUR400.0

Montant Total: EUR1500.0


Nous vous rappelons que vous pouvez maintenant proc?der au paiement par car=
te de cr?dit via Saferpay. Veuillez cliquer sur le lien ci-dessous et suivr=
e les instructions:


Nous vous ferons parvenir les articles command?s d?s r?ception de votre pai=
ement.=20

Pour toute question ou si vous souhaitez changer/annuler votre commande, ve=
uillez nous contacter ? ops@bla.int.

Cordialement,=20

.
250 OK
QUIT
221 listserv.bla.int So long, and thanks for all the fish
-----------------------------------------

*(3) - This is the method which I am using to send the email*
-----------------------------------------
public int sendEmail() 
     {
          //Set properties
          Properties properties = System.getProperties();
          properties.put("mail.smtp.host", this.mailSmtpServer);
          properties.put("mail.host", this.mailSmtpServer);
          properties.put("mail.transport.protocol", this.mailTransportProtocol);
          properties.put("mail.mime.charset", this.mailMimeCharSet);
          properties.put("mail.content.type", this.mailContentType);
          properties.put("mail.debug", "true");
          
          Session session = Session.getInstance(properties, null);
               
          //Create a MimeMessage object
          MimeMessage message = new MimeMessage(session);
          
          //Add FROM
          InternetAddress[] from = new InternetAddress[this.fromEmailAddresses.size()];
          for (int i=0;i<this.fromEmailAddresses.size();i++)
               from[i] = new InternetAddress(this.fromEmailAddresses.get(i));
          
          message.addFrom(from);
          
          //Add TO
          InternetAddress[] to = new InternetAddress[this.toEmailAddresses.size()];
          for (int i=0;i<this.toEmailAddresses.size();i++)
               to[i] = new InternetAddress(this.toEmailAddresses.get(i));
          
          message.setRecipients(Message.RecipientType.TO, to);
          
          if (replyEmailAddresses != null && replyEmailAddresses.size() > 0)
               {
               //Add REPLY-TO
               InternetAddress[] replyto = new InternetAddress[this.replyEmailAddresses.size()];
               for (int i=0;i<this.replyEmailAddresses.size();i++)
                    replyto[i] = new InternetAddress(this.replyEmailAddresses.get(i));
               
               message.setReplyTo(replyto);
               }
          
          //Set subject
          message.setSubject(this.emailSubject, this.mailMimeCharSet);
          
          //Set the email body and
          //Header "Content-Transfer-Encoding" attribute (otherwise, it takes the devault set by the JVM)
          message.setText(this.emailBody, this.mailMimeCharSet);
          message.setHeader("Content-Transfer-Encoding", "quoted-printable");     //"quoted-printable" / "7bit" / "8bit"
          message.setHeader("Content-Type", "text/plain; charset=\"UTF-8\"");
          
          //Save all attributes into the message object
          message.saveChanges();

          //This is the line responsible for dumping the contents of the email body in the log file shown in (1)
          logger.info("Email Contents:\n\n" + this.emailBody);
          
          //Send the message
          Transport.send(message);
          
          //Return success
          return 0;
     }
-----------------------------------------
  • 1. Re: [HELP]: Email sent from library running under Tomcat loses French chars
    bshannon Pro
    Currently Being Moderated
    The utf-8 message is encoded using the quoted-printable encoding, which is why the utf-8 characters don't appear in the log file.
  • 2. Re: [HELP]: Email sent from library running under Tomcat loses French chars
    901561 Newbie
    Currently Being Moderated
    Hi,

    Thanks for your reply....

    I am not sure that I am following you correctly....

    Even if I do not set that header attribute to "quoted-printable-encoding", I get the same result...

    The really strange thing is that the same exact piece of code (used to send the email) when run from a command-line (which means creating a separate JVM), runs correctly and the email is correctly received on my MS Outlook client (all the missing French chars are there - the output is exactly the same as the one dumped into the log file, as shown on (1) )...

    ....or perhaps I missunderstood you...? :)

    Thanks again,
    Gizmo
  • 3. Re: [HELP]: Email sent from library running under Tomcat loses French chars
    bshannon Pro
    Currently Being Moderated
    I don't know how the output in (1) is getting written to the log file, but it doesn't appear to come from the JavaMail debug output.

    If you write out the text from the message body yourself, you'll be writing out the unencoded Unicode characters, which gives you what you expect to see.

    When turning the content into a MIME message, JavaMail chooses an appropriate encoding for you, which is why it's using quoted-printable encoding. MIME messages are normally restricted to ASCII characters.
  • 4. Re: [HELP]: Email sent from library running under Tomcat loses French chars
    901561 Newbie
    Currently Being Moderated
    Hi,

    Thanks again for your help...

    I am currently dumping the generated email into the log with log4j (had to set a special encoding attribute to UTF-8)...

    Well, actually I decided to use quoted-printable encoding, instead of 7-bit or 8-bit...but it doesn't produce the expected result either...

    So what do you think that I am doing wrong? or how do you think, I might be able to solve this issue?

    Many thanks again for your help!

    Cheers,
    Gizmo
  • 5. Re: [HELP]: Email sent from library running under Tomcat loses French chars
    bshannon Pro
    Currently Being Moderated
    I've lost track of what it is you're trying to accomplish.

    I thought you were just trying to understand why you were getting different content in different places, which I explained.

    What is the problem you're trying to solve? What do you want the end result to be?
  • 6. Re: [HELP]: Email sent from library running under Tomcat loses French chars
    901561 Newbie
    Currently Being Moderated
    Hi again,

    Was getting worried about the Oracle Forums not coming back to life again...:)

    Anyway, my problem currently is that I want to receive the same output in my email box, as shown in the tomcat's logfile:

    *[1]*
    -----
    Cher Utilisateur,

    Vous avez commandé auprès d’EUMETSAT les articles suivants:

    Description, Quantité, Prix Unitaire

    - Meteosat Key Unit(MKU), 1 , EUR700.0
    - Meteosat Station Key Unit(SKU), 1 , EUR400.0
    - Metop Direct Readout Station Key Unit(SKU), 1 , EUR400.0

    Montant Total: EUR1500.0

    Nous vous rappelons que vous pouvez maintenant procéder au paiement par carte de crédit via Saferpay. Veuillez cliquer sur le lien ci-dessous et suivre les instructions:

    Nous vous ferons parvenir les articles commandés dès réception de votre paiement.

    Pour toute question ou si vous souhaitez changer/annuler votre commande, veuillez nous contacter à ops@bla.int.

    Cordialement,
    -----

    Now, the real catch is:

    1) On a different Solaris system, the email gets sent correctly (exactly as [1]) from the webapp running under a Tomcat instance (let's say Solaris-DEV). It is shown correctly in the Tomcat's log and also on the email inbox.

    2) On the Solaris-VAL system, the email is generated correctly (the email [1] gets correctly dumped in the Tomcat's log file), but when it is sent, the JavaMail API in Debug mode shows the french chars as a single "?" and that is also what shows up in the email received on my MS Outlook inbox.

    3) However, if I run exactly the same java method (shown above in my first post) from the JAR library as a standalone application (instead of within a Tomcat container), the email is generated correctly and also correctly received on my MS Outlook inbox (by "correctly" I mean, all French chars are displayed correctly).

    So what is currently driving me mad is the fact that the exact same Java code (compiled and bundled inside a JAR file) responsible for sending the email using the JavaMail API, seems to generate and send the email correctly when it runs as a standalone application on its own JVM, but when bundled with the webapplication running under Tomcat, the email seems to be correctly generated on the log, but then the French chars are lost (as shown by the JavaMail API DEBUG and also on the email received). - This happens on one of the Solaris systems ONLY.

    However, on the other Solaris system (Solaris-DEV), the webapplication running under tomcat generates the email correctly, the JavaMail DEBUG displays the email correctly and the email is correctly received on the mail inbox as well!

    Do you have any clue or suggestions of how can I figure out what's going on?

    Many thanks again for all your support
    Gizmo
  • 7. Re: [HELP]: Email sent from library running under Tomcat loses French chars
    bshannon Pro
    Currently Being Moderated
    It's a charset problem.

    The Tomcat logfile output is not coming from JavaMail, it's coming from your application
    simply writing out the string that you use as the main message content, right?

    What you need to compare is the MIME message content produced by JavaMail, which
    you can see in JavaMail's debug output, or which you can write out yourself using the
    Message.writeTo method.

    Also, make sure you're using the same version of JavaMail in all cases. The version of
    Tomcat you're using may be bundled with a different version of JavaMail, or may be
    picking up JavaMail from some different location.

    You can also check that the LANG and related LC_* environment variables are set the
    same in all cases.
  • 8. Re: [HELP]: Email sent from library running under Tomcat loses French chars
    901561 Newbie
    Currently Being Moderated
    Hi!

    (Bill Shannon, I presume?) :)

    Thanks for taking the time to read my cry for help :)

    You're right. I am currently outputting the contents of the email twice:
    a) By writing it directly into the log file, before passing it onto JavaMail - The email is printed correctly in this case.
    b) Also by putting the JavaMail DEBUG flag to true - This one prints the email with French chars as "?"

    Some questions though:

    1) I can find multiple mail.jar files in the system...currently using JavaMail 1.4.4 (found inside the /WEB-INF/lib/ of my webapp) but had the same issues with V1.4.0.

    It seems that the system has multiple other versions installed on other webapps (v1.3.1) and a really old version (JavaMail v1.2) found on /usr/apache/tomcat/common/lib/mail.jar - I am not sure whether this JavaMail 1.2 jar library is being used at all, since it is found on a old Tomcat installation directory

    In the event that it is used...do you know if are there any known issues with old JavaMail versions and V1.2 in particular?

    2) Also, does the JavaMail API rely on LANG and related LC_* variables?

    3) What do you mean by MIME comparison? :)

    Many thanks again,
    Cheers,
    Gizmo
  • 9. Re: [HELP]: Email sent from library running under Tomcat loses French chars
    bshannon Pro
    Currently Being Moderated
    Gizmo wrote:
    Hi!

    (Bill Shannon, I presume?) :)
    Yes.
    Thanks for taking the time to read my cry for help :)

    You're right. I am currently outputting the contents of the email twice:
    a) By writing it directly into the log file, before passing it onto JavaMail - The email is printed correctly in this case.
    b) Also by putting the JavaMail DEBUG flag to true - This one prints the email with French chars as "?"
    That shouldn't happen.
    Some questions though:

    1) I can find multiple mail.jar files in the system...currently using JavaMail 1.4.4 (found inside the /WEB-INF/lib/ of my webapp) but had the same issues with V1.4.0.

    It seems that the system has multiple other versions installed on other webapps (v1.3.1) and a really old version (JavaMail v1.2) found on /usr/apache/tomcat/common/lib/mail.jar - I am not sure whether this JavaMail 1.2 jar library is being used at all, since it is found on a old Tomcat installation directory
    Yes, it probably is being used.
    In the event that it is used...do you know if are there any known issues with old JavaMail versions and V1.2 in particular?
    There are tons of issues in older versions, see the CHANGES.txt file that comes with the latest version for a list.
    I don't remember whether any of those issues might be causing your problem, but it's best to eliminate it as a
    possibility by making sure that you're using the latest version.
    2) Also, does the JavaMail API rely on LANG and related LC_* variables?
    No, but the JDK does, and JavaMail uses the JDK to do all the charset conversions.
    3) What do you mean by MIME comparison? :)
    If you don't know what MIME is, see the JavaMail FAQ.

    I mean that you should compare the raw MIME versions of the messages to see what's different
    between the cases that work and the cases that don't work. Given the same message content,
    the raw MIME message content should be the same, except for the Message-ID.
  • 10. Re: [HELP]: Email sent from library running under Tomcat loses French chars
    901561 Newbie
    Currently Being Moderated
    Hi again,

    I followed your suggestion and dumped into the Log file:

    1) The generated email, before creating a MimeMessage object and setting its content
    2) The contents from the MimeMessage using the MimeMessage.writeTo(outputstream)
    3) Running the JavaMail in Debug mode also produced the expected output into the log

    One interesting finding though! The "content-transfer-encoding" header, which correctly shows up as in the DEV environment (where the email is printed correctly):

    Content-Transfer-Encoding: quoted-printable

    Whereas in the VAL environment (where the email is generated correctly but French characters are lost), the Mime header shows up as:

    Content-Transfer-Encoding: 7bit

    This is strange, I am using the exact same Java code, where the header is currently set to (hard-coded to "quoted-printable"), but the Content-Transfer-Encoding header appears as 7bit - in the MimeMessage dump and also in the JavaMail DEBUG?

    I'm a bit lost..:(

    Thanks again,
    Gizmo
  • 11. Re: [HELP]: Email sent from library running under Tomcat loses French chars
    bshannon Pro
    Currently Being Moderated
    You should never need to set the Content-Transfer-Encoding header yourself; JavaMail will choose an
    appropriate value based on the content.

    Did you fix it to use the same version of JavaMail in both environments?

    Are you using the same version of the JDK in both environments?

    Did you check the LANG and LC_* environment variables?

    If all those things are the same in both environments, you need to compare the content of the strings
    you're using for the email body. Dump out the integer value of each Unicode character in the string
    and compare between the two environments.
  • 12. Re: [HELP]: Email sent from library running under Tomcat loses French chars
    901561 Newbie
    Currently Being Moderated
    Hi,

    Actually, I changed the order by which I was setting the headers, so now I am setting the "Content-Transfer-Encoding" to "8bit" and "Content-Type" AFTER calling the "message.setSubject()" and "message.setText()", and the JavaMail Debug message now correclty shows the same "Content-Transfer-Encoding" value on both environments.

    I have also removed the call to "message.saveChanges()" from the code...

    With regard to your suggestions:

    1) I have just made sure that the JavaMail library's exactly the same (V1.4.4) in the 2 environments.

    2) The Java version which is used by Tomcat on the two environments is slightly different, but I think it's not problematic:
    - JDK 1.6.0_11 on DEV environment, JDK 1.6.0_29-b11 on VAL

    3) when typing "locale" on the command prompt, both systems show exactly the following env variables/values:

    LANG=
    LC_CTYPE="C"
    LC_NUMERIC="C"
    LC_TIME="C"
    LC_COLLATE="C"
    LC_MONETARY="C"
    LC_MESSAGES="C"
    LC_ALL=

    4) Going to dump the unicode char for each of the generated emails on both environments to see if any conclusion can be reached....

    Thanks again,
    Gizmo
  • 13. Re: [HELP]: Email sent from library running under Tomcat loses French chars
    901561 Newbie
    Currently Being Moderated
    Hi again,

    I have just dumped the ASCII contents of the the Generated Email (before being converted into a Mime Message) and afterwards (after calling setText() and getting the contents of the MimeMessage object)...

    This was the result on the VAL (problematic system).

    a) Before:
    http://www2.uninova.pt/~nv/EOP_GeneratedEmail_CharCodes.txt

    b) After object Mime Conversion:
    http://www2.uninova.pt/~nv/EOP_MimeMessagesEmailCharCodes.txt

    The following pairs of char code cab ne found on:

    a)

    É 201
    Ú 218
    é 233
    è 232
    ’ 8217

    b) All these chars appear as ? 63 on the String which I get from the MimeMessage object

    I'm getting the contents of the MimeMessage object with the following code:
    public static String getString(MimeMessage mesg)
    {
    ByteArrayOutputStream outBytes = new ByteArrayOutputStream();
    
    try
    {
    mesg.writeTo(outBytes);
    String theMimeMessage = new String(outBytes.toString("UTF-8"));
    
    return theMimeMessage ;
    }
    catch (Exception e)
    {
    lgger.error("ERROR: " + e.getMessage());
    return "";
    }
    }
    On the DEV everything was ok, I mean, the following chars:
    É 201
    Ú 218
    é 233
    è 232
    ’ 8217

    appeared correctly on the generated email String and after retrieval from the contents of the Mime Message...

    ...could I be dumping the generated email and Mime Contents wrongly?

    Thanks again for all your kind support.

    Cheers,
    Nuno

    PS: I am actually also printing the contents of the generated email (before being converted into part of the Mime Message object) in here:
    http://veoportal.eumetsat.int/userMgmt/testmail.jsp
  • 14. Re: [HELP]: Email sent from library running under Tomcat loses French chars
    bshannon Pro
    Currently Being Moderated
    The generated email should never have those utf-8 characters, it should only have ASCII characters.
    I see that the generated email has the Content-Transfer-Encoding set to 8bit. JavaMail won't normally do that.
    Are you sure you're using the Sun version of JavaMail and not the Gnu version of JavaMail?

    The decimal value 201 is hex c9, which is the iso-8859-1character É.

    Try this test program on your machines and see what you get:
    import javax.mail.*;
    import javax.mail.internet.*;
    import java.util.Properties;
    
    public class tc {
        public static void main(String args[]) throws Exception {
            Session session = Session.getInstance(new Properties());
            MimeMessage m = new MimeMessage(session);
            m.setText("test this utf-8 character: \u00c9\r\n", "utf-8");
            m.writeTo(System.out);
        }
    }
    You should get something like this:
    Message-ID: <31476927.0.1323892986332.JavaMail.shannon@datsunx>
    MIME-Version: 1.0
    Content-Type: text/plain; charset=utf-8
    Content-Transfer-Encoding: quoted-printable
    
    test this utf-8 character: =C3=89
1 2 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points