Go Back   Gaming Gutter > Non-Gaming > Programming > Source Code


Source Code - Have a source code/project files you want to post? Do so here.

» Site Navigation
» Home
» FAQ
» Log in
User Name:

Password:

Not a member yet?
Register Now!
» Advertisement
» Recent Threads
Go to first new post MotM Nomination Thread #1
Yesterday 08:08 PM
by nomhak
Last post by nomhak
Today 09:32 PM
13 Replies, 105 Views
Go to first new post What's up?
Today 09:31 PM
by Roxlo
Last post by Roxlo
Today 09:31 PM
0 Replies, 1 Views
Go to first new post Lending Sneezles Pet
Today 09:26 PM
by Judle
Last post by Judle
Today 09:26 PM
0 Replies, 2 Views
Go to first new post Where should I get a...
Yesterday 09:44 AM
Last post by virusdetected
Today 09:25 PM
6 Replies, 49 Views
Go to first new post Consumption
Today 09:23 PM
by Axed
Last post by Axed
Today 09:23 PM
0 Replies, 3 Views
Reply
 
LinkBack Thread Tools Display Modes

 Java URL class w/ GZip, Deflate & cookie support
Old 04-08-2007, 09:18 PM   #1 (permalink)
sockopen
Guest

 
Posts: n/a
GPoints: 0 [Check]
iTrader: / %
Java URL class w/ GZip, Deflate & cookie support

Code:
import java.io.InputStream;
import java.io.OutputStreamWriter;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.zip.GZIPInputStream;
import java.util.zip.Inflater;
import java.util.zip.InflaterInputStream;

import java.io.IOException;
import java.net.UnknownHostException;

public class HTTPWrapper {
	
	public static String strCookies;
	private static String strHTML;
	
	public static void initHTTPWrapper() {
		strCookies = "";
	}
	
	public static String Request(String URL, String PostData, String Referer) {
		String method; if (PostData != "") method = "POST"; else method = "GET";
		try{
			URL url = new URL("http://" + URL);
			HttpURLConnection http = (HttpURLConnection) url.openConnection();
			http.setFollowRedirects(false);
			http.setRequestMethod(method);
			http.setRequestProperty("Host", url.getHost());
			http.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3");
			http.setRequestProperty("Accept", "text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5");
			http.setRequestProperty("Accept-Language", "en-us,en;q=0.5");
			http.setRequestProperty("Accept-Encoding", "gzip,deflate");
			http.setRequestProperty("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.7");
			http.setRequestProperty("Keep-Alive", "300");
			http.setRequestProperty("Connection", "keep-alive");
			if (Referer != "0")
				http.setRequestProperty("Referer", Referer);
			if (strCookies != "")
				http.setRequestProperty("Cookie", strCookies);
			if (method == "POST") {
				http.setDoOutput(true);
				http.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
				http.setRequestProperty("Content-Length", "" + PostData.length());
				OutputStreamWriter writePost = new OutputStreamWriter(http.getOutputStream());
				writePost.write(PostData);
				writePost.flush();
			}
			http.connect();
			String encoding = http.getContentEncoding();
			InputStream receiving = null;
			if (encoding.equalsIgnoreCase("gzip"))
				receiving = new GZIPInputStream(http.getInputStream());
			else if (encoding.equalsIgnoreCase("deflate"))
				receiving = new InflaterInputStream(http.getInputStream(), new Inflater(true));
			else
				receiving = http.getInputStream();
			int intReceived; StringBuffer strBuffer = new StringBuffer();
			while((intReceived = receiving.read()) != -1) {
				strBuffer.append((char) intReceived);
			}
			receiving.close();
			strHTML = strBuffer.toString();
			strBuffer.delete(0, strBuffer.length()); strBuffer.append(strCookies);
			if (strCookies.length() > 3) strBuffer.append("; ");
			for (int i=0;; i++) {
				if (http.getHeaderFieldKey(i) == null && http.getHeaderField(i) == null) {
					break;
				} else if ("Set-Cookie".equalsIgnoreCase(http.getHeaderFieldKey(i))) {
					String[] fields = http.getHeaderField(i).split(";\\s*");
					String[] keyvalue = fields[0].split("=");
					if (strBuffer.indexOf(keyvalue[0] + "=") == -1)
						strBuffer.append(keyvalue[0] + "=" + keyvalue[1] + "; "); 
					else
						strBuffer.replace(strBuffer.indexOf(keyvalue[0]), strBuffer.indexOf(";", strBuffer.indexOf(keyvalue[0])), keyvalue[0] + "=" + keyvalue[1]);
				}
			}
			if (strBuffer.indexOf("; ", strBuffer.length() - 3) != -1)
				strBuffer.delete(strBuffer.length() - 2, strBuffer.length());
			strCookies = strBuffer.toString();
		} catch (Exception e) {}
		return strHTML;
	}
}
You should initiate the HTTPWrapper before Request'ing, which initiates the Cookies and Proxy variables and will prevent our class from throwing an exception. HTTPWrapper.initHTTPWrapper;

Proxy support and everything is all easily done with the HttpURLConnection ('http' in this code snippet I wrote).

Usage (GET):
HTTPWrapper.Request("www.neopets.com", "", "www.Referer.com");

Usage (POST):
HTTPWrapper.Request("www.neopets.com/login.phtml", "Username=Password=", "www.Referer.com");
  Reply With Quote

 
Old 04-21-2007, 05:04 PM   #2 (permalink)
Executor89
Guest

 
Posts: n/a
GPoints: 0 [Check]
iTrader: / %
Sorry for asking, but what does this program do?
  Reply With Quote

 
Old 04-22-2007, 02:37 PM   #3 (permalink)
sockopen
Guest

 
Posts: n/a
GPoints: 0 [Check]
iTrader: / %
Quote:
Originally Posted by Executor89 View Post
Sorry for asking, but what does this program do?
It's not as much as a program as it is a function / class. You can use this class similarly to an 'HTTPWrapper', which, simply requests and returns the contents of a website. ie;

HTTPWrapper.Request('google.com', '', '');

will return the following:

Code:
<html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><title>Google</title><style><!--
body,td,a,p,.h{font-family:arial,sans-serif}
.h{font-size:20px}
.h{color:#3366cc}
.q{color:#00c}
--></style>
<script>
<!--
function sf(){document.f.q.focus();}
window.clk=function(b,c,d,h,i,j){if(document.images){var a=window.encodeURIComponent?encodeURIComponent:escape,e="",f="",g="";if(b){e="&url="+a(b.replace(/#.*/,"")).replace(/\+/g,"%2B")}if(c){f="&oi="+a(c)}if(d){g="&cad="+a(d)}(new Image).src="/url?sa=T"+f+g+"&ct="+a(h)+"&cd="+a(i)+e+"&ei=YccrRoeIKoyK_AHK9cW_DQ"+j}return true};// -->
</script>
</head><body bgcolor=#ffffff text=#000000 link=#0000cc vlink=#551a8b alink=#ff0000 onload="sf();if(document.images){new Image().src='/images/nav_logo3.png'}" topmargin=3 marginheight=3><center><div align=right id=guser style="font-size:84%;padding-bottom:4px" width=100%><nobr><a href="/url?sa=p&pref=ig&pval=3&q=http://www.google.ca/ig%3Fhl%3Den&usg=AFrqEzfLTpEgdqdTMF8AvHxexi4WjZiSkw" onmousedown="return clk('/url?sa=p&pref=ig&pval=3&q=http://www.google.ca/ig%3Fhl%3Den&usg=AFrqEzfLTpEgdqdTMF8AvHxexi4WjZiSkw','promos','hppphnu:def','pro','1','')">Personalize this page</a>&nbsp;|&nbsp;<a href="https://www.google.com/accounts/Login?continue=http://www.google.ca/&hl=en">Sign in</a></nobr></div><a href="/search?q=earth+day"><img src=/logos/earthday07.gif width=330 height=125 border=0 alt="Earth Day" title="Earth Day"></a><br><br><form action="/search" name=f><script defer><!--
function qs(el){if(window.RegExp&&window.encodeURIComponent){var ue=el.href,qe=encodeURIComponent(document.f.q.value);if(ue.indexOf("q=")!=-1){el.href=ue.replace(new RegExp("q=[^&$]*"),"q="+qe);}else{el.href=ue+"&q="+qe;}}return 1;}
//-->
</script><table border=0 cellspacing=0 cellpadding=4><tr><td nowrap><font size=-1><b>Web</b>&nbsp;&nbsp;&nbsp;&nbsp;<a class=q href="http://images.google.ca/imghp?ie=UTF-8&oe=UTF-8&hl=en&tab=wi" onclick="return qs(this)">Images</a>&nbsp;&nbsp;&nbsp;&nbsp;<a class=q href="http://groups.google.ca/grphp?ie=UTF-8&oe=UTF-8&hl=en&tab=wg" onclick="return qs(this)">Groups</a>&nbsp;&nbsp;&nbsp;&nbsp;<a class=q href="http://news.google.ca/nwshp?ie=UTF-8&oe=UTF-8&hl=en&tab=wn" onclick="return qs(this)">News</a>&nbsp;&nbsp;&nbsp;&nbsp;<a class=q href="http://maps.google.ca/maps?ie=UTF-8&oe=UTF-8&hl=en&tab=wl" onclick="return qs(this)">Maps</a>&nbsp;&nbsp;&nbsp;&nbsp;<b><a href="/intl/en/options/" class=q>more&nbsp;&raquo;</a></b></font></td></tr></table><table cellpadding=0 cellspacing=0><tr valign=top><td width=25%>&nbsp;</td><td align=center nowrap><input name=hl type=hidden value=en><input maxlength=2048 name=q size=55 title="Google Search" value=""><br><input name=btnG type=submit value="Google Search"><input name=btnI type=submit value="I'm Feeling Lucky"></td><td nowrap width=25%><font size=-2>&nbsp;&nbsp;<a href=/advanced_search?hl=en>Advanced Search</a><br>&nbsp;&nbsp;<a href=/preferences?hl=en>Preferences</a><br>&nbsp;&nbsp;<a href=/language_tools?hl=en>Language Tools</a></font></td></tr><tr><td align=center colspan=3><font size=-1>Search: <input id=all type=radio name=meta value="" checked><label for=all> the web </label><input id=cty type=radio name=meta value="cr=countryCA"><label for=cty> pages from Canada </label></font></td></tr></table></form><br><font size=-1>Google.ca offered in: <a href="http://www.google.ca/fr">Français</a> </font><br><br><br><font size=-1><a href="/intl/en/ads/">Advertising&nbsp;Programs</a> - <a href="/services/">Business Solutions</a> - <a href="/intl/en/about.html">About Google</a> - <a href=http://www.google.com/ncr>Go to Google.com</a><span id=hp style="behavior:url(#default#homepage)"></span><script><!--
(function() {var a="http://www.google.ca/",b=document.getElementById("hp"),c=b.isHomePage(a);if(!c){document.write('<p><a href=/mgyhp.html onClick=document.getElementById("hp").setHomepage("'+a+'")>Make Google Your Homepage!</a>')};})();//-->
</script></font><p><font size=-2>&copy;2007 Google</font></p><style>#ietb{behavior:url(#default#userData);display:none;text-align:center;width:250px;position:absolute;top:2px;right:2px;border:1px solid #656565;font-weight:bold;background:#fff;padding:1px 0 4px 4px;font-family:arial}</style><div id=ietb><span id=close><a href=# onclick="return _cltbp()"><img src="images/close_sm.gif" width=12 height=12 border=0 align=right style="padding:2px"></a></span><br><p style=margin-top:15px;text-align:center;font-weight:bold><font size=-1>Get a Google-enhanced search box</font></p><img src=intl/en-GB/images/toolbar_sm.png><br><input type=button style=margin-top:12px value="Download Google Toolbar" onClick=_dwntbp();><font size=-3><br><br></font></div><script>window._settbp=function(){if(document){var a=document.getElementById("ietb");a.load("IsOnIE7tbPromo");if(a.getAttribute("display")==null){a.style.display="block";(new Image).src="/gen_204?oi=promos_vis&cad=hppwebie7tb:en-GB&atyp=i"}}};window._cltbp=function(){if(document){var a=document.getElementById("ietb");a.setAttribute("display","none");a.save("IsOnIE7tbPromo");a.style.display="none";(new Image).src="/gen_204?oi=promos&ct=remove&cad=hppwebie7tb:en-GB&sa=X";return false}};window._dwntbp=function(){if(document){(new Image).src="/gen_204?oi=promos&ct=download&cad=hppwebie7tb:en-GB&sa=X";document.location="/toolbar/intl/en-GB/webinstall.html#tbbrand=GFRD"}};_settbp();</script></center></body></html>
  Reply With Quote

 
Old 03-02-2008, 09:01 PM   #4 (permalink)
Full Member

unlimitedorb is offline
 
Join Date: Feb 2008
Posts: 25
GPoints: 29
iTrader: 0 / 0%
unlimitedorb Is gaining popularity
Rep Power: 3
Handling data sent from the server one char at a time is bad. What you want to do is buffer it, so one possible way would be to use Java's BufferedReader. Furthermore, if the application isn't multi-threaded you might want to go ahead and use StringBuilder instead of StringBuffer for optimal speed.

Example:
BufferedReader receivingBuffer = new BufferedReader(new InputStreamReader(receiving));
String line;
while((line = receivingBuffer.readLine()) != null) {
strBuffer.append(line.toCharArray());
}

Proxy support is easily tacked on like you said, but you might want to rethink handling cookies yourself as the code you wrote can easily break. A high point of Java is that there is a large repository to build on...use it ;) I use jCookie for cookie handling.

EDIT: I believe IOException encompasses UnknownHostException, so you don't have to catch UnknownHostException.

Last edited by unlimitedorb; 03-02-2008 at 09:31 PM.
  Reply With Quote
Reply

Bookmarks



Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Powered by vBadvanced CMPS v3.0 RC2

All times are GMT -7. The time now is 09:33 PM.


vBulletin skin developed by: eXtremepixels
vBCredits v1.4 Copyright ©2007 - 2008, PixelFX Studios
The contents of this webpage are copyright © 2006-2008 GamingGutter.com. All Rights Reserved.

Page generated in 0.09959412 seconds (100.00% PHP - 0% MySQL) with 17 queries