» Site Navigation | | | » Advertisement | | | » Recent Threads | | | What's up? Today 09:31 PM Today 09:31 PM 0 Replies, 1 Views | | | | | Consumption Today 09:23 PM Today 09:23 PM 0 Replies, 3 Views | |  |  | Java URL class w/ GZip, Deflate & cookie support |  |
04-08-2007, 09:18 PM
|
#1 (permalink)
| | | Java URL class w/ GZip, Deflate & cookie support Code: import java.io.InputStream;
import java.io.OutputStreamWriter;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.zip.GZIPInputStream;
import java.util.zip.Inflater;
import java.util.zip.InflaterInputStream;
import java.io.IOException;
import java.net.UnknownHostException;
public class HTTPWrapper {
public static String strCookies;
private static String strHTML;
public static void initHTTPWrapper() {
strCookies = "";
}
public static String Request(String URL, String PostData, String Referer) {
String method; if (PostData != "") method = "POST"; else method = "GET";
try{
URL url = new URL("http://" + URL);
HttpURLConnection http = (HttpURLConnection) url.openConnection();
http.setFollowRedirects(false);
http.setRequestMethod(method);
http.setRequestProperty("Host", url.getHost());
http.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3");
http.setRequestProperty("Accept", "text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5");
http.setRequestProperty("Accept-Language", "en-us,en;q=0.5");
http.setRequestProperty("Accept-Encoding", "gzip,deflate");
http.setRequestProperty("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.7");
http.setRequestProperty("Keep-Alive", "300");
http.setRequestProperty("Connection", "keep-alive");
if (Referer != "0")
http.setRequestProperty("Referer", Referer);
if (strCookies != "")
http.setRequestProperty("Cookie", strCookies);
if (method == "POST") {
http.setDoOutput(true);
http.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
http.setRequestProperty("Content-Length", "" + PostData.length());
OutputStreamWriter writePost = new OutputStreamWriter(http.getOutputStream());
writePost.write(PostData);
writePost.flush();
}
http.connect();
String encoding = http.getContentEncoding();
InputStream receiving = null;
if (encoding.equalsIgnoreCase("gzip"))
receiving = new GZIPInputStream(http.getInputStream());
else if (encoding.equalsIgnoreCase("deflate"))
receiving = new InflaterInputStream(http.getInputStream(), new Inflater(true));
else
receiving = http.getInputStream();
int intReceived; StringBuffer strBuffer = new StringBuffer();
while((intReceived = receiving.read()) != -1) {
strBuffer.append((char) intReceived);
}
receiving.close();
strHTML = strBuffer.toString();
strBuffer.delete(0, strBuffer.length()); strBuffer.append(strCookies);
if (strCookies.length() > 3) strBuffer.append("; ");
for (int i=0;; i++) {
if (http.getHeaderFieldKey(i) == null && http.getHeaderField(i) == null) {
break;
} else if ("Set-Cookie".equalsIgnoreCase(http.getHeaderFieldKey(i))) {
String[] fields = http.getHeaderField(i).split(";\\s*");
String[] keyvalue = fields[0].split("=");
if (strBuffer.indexOf(keyvalue[0] + "=") == -1)
strBuffer.append(keyvalue[0] + "=" + keyvalue[1] + "; ");
else
strBuffer.replace(strBuffer.indexOf(keyvalue[0]), strBuffer.indexOf(";", strBuffer.indexOf(keyvalue[0])), keyvalue[0] + "=" + keyvalue[1]);
}
}
if (strBuffer.indexOf("; ", strBuffer.length() - 3) != -1)
strBuffer.delete(strBuffer.length() - 2, strBuffer.length());
strCookies = strBuffer.toString();
} catch (Exception e) {}
return strHTML;
}
} You should initiate the HTTPWrapper before Request'ing, which initiates the Cookies and Proxy variables and will prevent our class from throwing an exception. HTTPWrapper.initHTTPWrapper;
Proxy support and everything is all easily done with the HttpURLConnection ('http' in this code snippet I wrote).
Usage (GET):
HTTPWrapper.Request("www.neopets.com", "", "www.Referer.com");
Usage (POST):
HTTPWrapper.Request("www.neopets.com/login.phtml", "Username=Password=", "www.Referer.com"); | |
| |  |
04-21-2007, 05:04 PM
|
#2 (permalink)
| | | Sorry for asking, but what does this program do? | |
| |  | |  |
04-22-2007, 02:37 PM
|
#3 (permalink)
| | | Quote:
Originally Posted by Executor89 Sorry for asking, but what does this program do? | It's not as much as a program as it is a function / class. You can use this class similarly to an 'HTTPWrapper', which, simply requests and returns the contents of a website. ie;
HTTPWrapper.Request('google.com', '', '');
will return the following: Code: <html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><title>Google</title><style><!--
body,td,a,p,.h{font-family:arial,sans-serif}
.h{font-size:20px}
.h{color:#3366cc}
.q{color:#00c}
--></style>
<script>
<!--
function sf(){document.f.q.focus();}
window.clk=function(b,c,d,h,i,j){if(document.images){var a=window.encodeURIComponent?encodeURIComponent:escape,e="",f="",g="";if(b){e="&url="+a(b.replace(/#.*/,"")).replace(/\+/g,"%2B")}if(c){f="&oi="+a(c)}if(d){g="&cad="+a(d)}(new Image).src="/url?sa=T"+f+g+"&ct="+a(h)+"&cd="+a(i)+e+"&ei=YccrRoeIKoyK_AHK9cW_DQ"+j}return true};// -->
</script>
</head><body bgcolor=#ffffff text=#000000 link=#0000cc vlink=#551a8b alink=#ff0000 onload="sf();if(document.images){new Image().src='/images/nav_logo3.png'}" topmargin=3 marginheight=3><center><div align=right id=guser style="font-size:84%;padding-bottom:4px" width=100%><nobr><a href="/url?sa=p&pref=ig&pval=3&q=http://www.google.ca/ig%3Fhl%3Den&usg=AFrqEzfLTpEgdqdTMF8AvHxexi4WjZiSkw" onmousedown="return clk('/url?sa=p&pref=ig&pval=3&q=http://www.google.ca/ig%3Fhl%3Den&usg=AFrqEzfLTpEgdqdTMF8AvHxexi4WjZiSkw','promos','hppphnu:def','pro','1','')">Personalize this page</a> | <a href="https://www.google.com/accounts/Login?continue=http://www.google.ca/&hl=en">Sign in</a></nobr></div><a href="/search?q=earth+day"><img src=/logos/earthday07.gif width=330 height=125 border=0 alt="Earth Day" title="Earth Day"></a><br><br><form action="/search" name=f><script defer><!--
function qs(el){if(window.RegExp&&window.encodeURIComponent){var ue=el.href,qe=encodeURIComponent(document.f.q.value);if(ue.indexOf("q=")!=-1){el.href=ue.replace(new RegExp("q=[^&$]*"),"q="+qe);}else{el.href=ue+"&q="+qe;}}return 1;}
//-->
</script><table border=0 cellspacing=0 cellpadding=4><tr><td nowrap><font size=-1><b>Web</b> <a class=q href="http://images.google.ca/imghp?ie=UTF-8&oe=UTF-8&hl=en&tab=wi" onclick="return qs(this)">Images</a> <a class=q href="http://groups.google.ca/grphp?ie=UTF-8&oe=UTF-8&hl=en&tab=wg" onclick="return qs(this)">Groups</a> <a class=q href="http://news.google.ca/nwshp?ie=UTF-8&oe=UTF-8&hl=en&tab=wn" onclick="return qs(this)">News</a> <a class=q href="http://maps.google.ca/maps?ie=UTF-8&oe=UTF-8&hl=en&tab=wl" onclick="return qs(this)">Maps</a> <b><a href="/intl/en/options/" class=q>more »</a></b></font></td></tr></table><table cellpadding=0 cellspacing=0><tr valign=top><td width=25%> </td><td align=center nowrap><input name=hl type=hidden value=en><input maxlength=2048 name=q size=55 title="Google Search" value=""><br><input name=btnG type=submit value="Google Search"><input name=btnI type=submit value="I'm Feeling Lucky"></td><td nowrap width=25%><font size=-2> <a href=/advanced_search?hl=en>Advanced Search</a><br> <a href=/preferences?hl=en>Preferences</a><br> <a href=/language_tools?hl=en>Language Tools</a></font></td></tr><tr><td align=center colspan=3><font size=-1>Search: <input id=all type=radio name=meta value="" checked><label for=all> the web </label><input id=cty type=radio name=meta value="cr=countryCA"><label for=cty> pages from Canada </label></font></td></tr></table></form><br><font size=-1>Google.ca offered in: <a href="http://www.google.ca/fr">Français</a> </font><br><br><br><font size=-1><a href="/intl/en/ads/">Advertising Programs</a> - <a href="/services/">Business Solutions</a> - <a href="/intl/en/about.html">About Google</a> - <a href=http://www.google.com/ncr>Go to Google.com</a><span id=hp style="behavior:url(#default#homepage)"></span><script><!--
(function() {var a="http://www.google.ca/",b=document.getElementById("hp"),c=b.isHomePage(a);if(!c){document.write('<p><a href=/mgyhp.html onClick=document.getElementById("hp").setHomepage("'+a+'")>Make Google Your Homepage!</a>')};})();//-->
</script></font><p><font size=-2>©2007 Google</font></p><style>#ietb{behavior:url(#default#userData);display:none;text-align:center;width:250px;position:absolute;top:2px;right:2px;border:1px solid #656565;font-weight:bold;background:#fff;padding:1px 0 4px 4px;font-family:arial}</style><div id=ietb><span id=close><a href=# onclick="return _cltbp()"><img src="images/close_sm.gif" width=12 height=12 border=0 align=right style="padding:2px"></a></span><br><p style=margin-top:15px;text-align:center;font-weight:bold><font size=-1>Get a Google-enhanced search box</font></p><img src=intl/en-GB/images/toolbar_sm.png><br><input type=button style=margin-top:12px value="Download Google Toolbar" onClick=_dwntbp();><font size=-3><br><br></font></div><script>window._settbp=function(){if(document){var a=document.getElementById("ietb");a.load("IsOnIE7tbPromo");if(a.getAttribute("display")==null){a.style.display="block";(new Image).src="/gen_204?oi=promos_vis&cad=hppwebie7tb:en-GB&atyp=i"}}};window._cltbp=function(){if(document){var a=document.getElementById("ietb");a.setAttribute("display","none");a.save("IsOnIE7tbPromo");a.style.display="none";(new Image).src="/gen_204?oi=promos&ct=remove&cad=hppwebie7tb:en-GB&sa=X";return false}};window._dwntbp=function(){if(document){(new Image).src="/gen_204?oi=promos&ct=download&cad=hppwebie7tb:en-GB&sa=X";document.location="/toolbar/intl/en-GB/webinstall.html#tbbrand=GFRD"}};_settbp();</script></center></body></html> | |
| |  |  | |  |
03-02-2008, 09:01 PM
|
#4 (permalink)
| Full Member
Join Date: Feb 2008 Posts: 25
GPoints: 29 Rep Power: 3 | Handling data sent from the server one char at a time is bad. What you want to do is buffer it, so one possible way would be to use Java's BufferedReader. Furthermore, if the application isn't multi-threaded you might want to go ahead and use StringBuilder instead of StringBuffer for optimal speed.
Example:
BufferedReader receivingBuffer = new BufferedReader(new InputStreamReader(receiving));
String line;
while((line = receivingBuffer.readLine()) != null) {
strBuffer.append(line.toCharArray());
}
Proxy support is easily tacked on like you said, but you might want to rethink handling cookies yourself as the code you wrote can easily break. A high point of Java is that there is a large repository to build on...use it ;) I use jCookie for cookie handling.
EDIT: I believe IOException encompasses UnknownHostException, so you don't have to catch UnknownHostException.
Last edited by unlimitedorb; 03-02-2008 at 09:31 PM.
| |
| |  |  | |
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | | | | Thread Tools | | | | Display Modes | Linear Mode |
Posting Rules
| You may not post new threads You may not post replies You may not post attachments You may not edit your posts HTML code is Off | | | |