1 2 3 Previous Next 35 Replies Latest reply: Mar 6, 2012 11:44 PM by EJP Go to original post RSS
      • 15. Re: How to use HttpURLConnection POST data to web server?
        EJP
        Well that now comes down to what gimbal2 suggested. You need to check that you have the correct variable names, and correct contents for those variables.
        • 16. Re: How to use HttpURLConnection POST data to web server?
          902241
          I print 'data' variable and it gives me exactly what I wanted:

          UF=SC&Localidade=Florianopolis
          • 17. Re: How to use HttpURLConnection POST data to web server?
            DrClap
            I just went to the web site that you're trying to scrape, and tried the example you suggested. I used HttpFox to observer the HTTP traffic and there was only one POST request. That's the one you're trying to replicate. However when I looked at the contents of the POST request, there were actually 7 parameters being sent. Looks like you're only sending the first two; the other 5 must be hidden parameters but I expect if the server doesn't get them then it won't handle the request correctly.
            • 18. Re: How to use HttpURLConnection POST data to web server?
              902241
              Well,


              I added the missing 5 hidden parameters and here's the new code:
                bombearURL ("http://www.buscacep.correios.com.br/servicos/dnec/menuAction.do?Metodo=menuFaixaCep", 
                                 new Par("UF", "SC"), 
                                 new Par("Localidade", "Florianopolis"),
                                 new Par("cfm", "1"),
                                 new Par("Metodo", "listaFaixaCEP"),
                                 new Par("TipoConsulta", "faixaCep"),
                                 new Par("StartRow", "1"),
                                 new Par("EndRow", "10")
                       );
                
              At the end I print the parameters string: UF=SC&Localidade=Florianopolis&cfm=1&Metodo=listaFaixaCEP&TipoConsulta=faixaCep&StartRow=1&EndRow=10

              And the response code was 200.

              However I wasn't still able to post data to the form. The server returned the same initial page.
              • 19. Re: How to use HttpURLConnection POST data to web server?
                EJP
                Did you notice you now have 'Metodo' twice? Remove it from the URL.
                • 20. Re: How to use HttpURLConnection POST data to web server?
                  902241
                  Yeah, I just noticed that.

                  However when I deleted the variable 'Metodo' from the URL the server returned me a 405 Method Not Allowed response code and the program raised an java.io.IOException

                  When I tried to change the value to 'listaFaixaCEP' - the value of the hidden 'Metodo' - I also got a 405 response and the program raised an java.io.IOException as well.
                  • 21. Re: How to use HttpURLConnection POST data to web server?
                    EJP
                    That's pretty odd because the parameter value is ambiguous. Some Web frameworks wouldn't deliver both values.

                    Anyway you are now deep into how the webapp itself works. Good luck.
                    • 22. Re: How to use HttpURLConnection POST data to web server?
                      902241
                      Well, at least I could obtain the response for
                      bombearURL("http://codepad.org/", 
                                          new Par("lang", "PHP"),
                                          new Par("code", "<?php echo \"Deu certo\"; ?>"),
                                          new Par("private", "False"),
                                          new Par("run", "True"));
                      • 23. Re: How to use HttpURLConnection POST data to web server?
                        DrClap
                        HttpFox also shows that the site is using cookies. Not surprisingly the cookie's name is JSESSIONID, since it's a Struts application at the back end. You ought to duplicate that part of the process too.
                        • 24. Re: How to use HttpURLConnection POST data to web server?
                          902241
                          As a matter of fact I did notice the JSESSIONID cookie whose path is /servicos/dnec

                          Am I supposed to fullfill it with another value?

                          How can I overcome this ?

                          Thank you!
                          • 25. Re: How to use HttpURLConnection POST data to web server?
                            EJP
                            You're probably supposed to login to the site somehow. See java.net.Authenticator.
                            • 26. Re: How to use HttpURLConnection POST data to web server?
                              902241
                              For sure! I was so sleepy yesterday I couldn´t think straight!

                              If you look at the home page there´s no place to login : [Home page|http://www.correios.com.br/]

                              or at the previously refered page either.
                              • 27. Re: How to use HttpURLConnection POST data to web server?
                                DrClap
                                899238 wrote:
                                As a matter of fact I did notice the JSESSIONID cookie whose path is /servicos/dnec

                                Am I supposed to fullfill it with another value?
                                No, you are supposed to use the cookie which the site gives you. To do that you "go" to the page where the online form is, by creating a URL for that page and getting the response from its connection. The response is just going to be the HTML for that page, so you can ignore that, but you should find you get a cookie back as well. That's the cookie you need to send with your POST request.

                                At least that's my guess. There isn't any authentication required, I've tried that page myself and got the response that I assume you were looking for. However I didn't try configuring my browser to reject cookies from the site, so I haven't actually tested what happens if the browser doesn't send the cookie.
                                • 28. Re: How to use HttpURLConnection POST data to web server?
                                  902241
                                  I added this snipet
                                  String headerfields = conn.getHeaderField(0);
                                               System.out.println(headerfields);
                                               System.out.println("---Start of headers---");
                                               int i = 1;
                                               while ((headerfields = conn.getHeaderField(i)) != null) {
                                               String key = conn.getHeaderFieldKey(i);
                                               System.out.println(((key==null) ? "" : key + ": ") + headerfields);
                                               i++;
                                                
                                               }
                                  and got this result:

                                  ---Start of headers---
                                  Date: Wed, 07 Mar 2012 00:09:26 GMT
                                  Server: Oracle-Application-Server-10g/10.1.2.0.2 Oracle-HTTP-Server
                                  Content-Location: http://www.buscacep.correios.com.br/servicos/dnec/WEB-INF/jsp/dnec.jsp
                                  Set-Cookie: JSESSIONID=c0a8321330d674920e9b0d054a8f8b2eab1b5ca160b4.e38QbheLb3yRci0Lbx0MbhiLchiPe6fznA5Pp7ftolbGmkTy; path=/servicos/dnec
                                  Cache-Control: private
                                  Keep-Alive: timeout=10, max=1000
                                  Connection: Keep-Alive
                                  Transfer-Encoding: chunked
                                  Content-Type: text/html;charset=ISO-8859-1



                                  As you can see JSESSIONID is already set. So where do I go from here?

                                  Thank you in advance.
                                  • 29. Re: How to use HttpURLConnection POST data to web server?
                                    EJP
                                    But is it set to the same value you got when you did GET on the page? That's what DrClap is describing.