Web Scraping with multipart/form-data post variables and share the same cookie between two urls on same website
In this post, i am going to share a code with you which i use use for Web Scraping with multipart/form-data post variables.
and share the same cookie between two urls on same website.
———— code ————————
private readonly Encoding encoding = Encoding.UTF8;
public HttpWebResponse GetResponseWithMultipartPost(string postUrl, string userAgent, Dictionary<string, object> postParameters, CookieContainer CC)
{
try
{
string formDataBoundary = “—————————–” + DateTime.Now.Ticks.ToString(“x”);
string contentType = “multipart/form-data; boundary=” + formDataBoundary;
byte[] formData = GetMultipartFormData(postParameters, formDataBoundary);
return PostForm(postUrl, userAgent, contentType, formData, CC);
}
catch (Exception ex)
{
string formDataBoundary = “—————————–” + DateTime.Now.Ticks.ToString(“x”);
string contentType = “multipart/form-data; boundary=” + formDataBoundary;
byte[] formData = GetMultipartFormData(postParameters, formDataBoundary);
Thread.Sleep(Config.ThreadSleepTime.ReWebRequest);//give the server a break
return GetResponseWithMultipartPost(postUrl, userAgent, postParameters, CC);
}
}
private byte[] GetMultipartFormData(Dictionary<string, object> postParameters, string boundary)
{
Stream formDataStream = new System.IO.MemoryStream();
foreach (var param in postParameters)
{
if (param.Value is FileParameter)
{
FileParameter fileToUpload = (FileParameter)param.Value;
// Add just the first part of this param, since we will write the file data directly to the Stream
string header = string.Format(“–{0}\r\nContent-Disposition: form-data; name=\”{1}\”; filename=\”{2}\”;\r\nContent-Type: {3}\r\n\r\n”,
boundary,
param.Key,
fileToUpload.FileName ?? param.Key,
fileToUpload.ContentType ?? “application/octet-stream”);
formDataStream.Write(encoding.GetBytes(header), 0, header.Length);
// Write the file data directly to the Stream, rather than serializing it to a string.
formDataStream.Write(fileToUpload.File, 0, fileToUpload.File.Length);
// Thanks to feedback from commenters, add a CRLF to allow multiple files to be uploaded
formDataStream.Write(encoding.GetBytes(“\r\n”), 0, 2);
}
else
{
string postData = string.Format(“–{0}\r\nContent-Disposition: form-data; name=\”{1}\”\r\n\r\n{2}\r\n”,
boundary,
param.Key,
param.Value);
formDataStream.Write(encoding.GetBytes(postData), 0, postData.Length);
}
}
// Add the end of the request
string footer = “\r\n–” + boundary + “–\r\n”;
formDataStream.Write(encoding.GetBytes(footer), 0, footer.Length);
// Dump the Stream into a byte[]
formDataStream.Position = 0;
byte[] formData = new byte[formDataStream.Length];
formDataStream.Read(formData, 0, formData.Length);
formDataStream.Close();
return formData;
}
private HttpWebResponse PostForm(string postUrl, string userAgent, string contentType, byte[] formData,CookieContainer CC)
{
HttpWebRequest request = WebRequest.Create(postUrl) as HttpWebRequest;
if (request == null)
{
throw new NullReferenceException(“request is not a http request”);
}
// Set up the request properties
request.Method = “POST”;
request.ContentType = contentType;
request.UserAgent = userAgent;
request.CookieContainer = CC;
request.ContentLength = formData.Length; // We need to count how many bytes we’re sending.
using (Stream requestStream = request.GetRequestStream())
{
// Push it out there
requestStream.Write(formData, 0, formData.Length);
requestStream.Close();
}
return request.GetResponse() as HttpWebResponse;
}
public class FileParameter
{
public byte[] File { get; set; }
public string FileName { get; set; }
public string ContentType { get; set; }
public FileParameter(byte[] file) : this(file, null) { }
public FileParameter(byte[] file, string filename) : this(file, filename, null) { }
public FileParameter(byte[] file, string filename, string contenttype)
{
File = file;
FileName = filename;
ContentType = contenttype;
}
}
// call the method.
Private void Callmethod()
{
// Generate post objects
Dictionary<string, object> postParameters = new Dictionary<string, object>();
postParameters.Add(“Field0”, “1”);
postParameters.Add(“Field1”, “2”);
// Create request and receive response
string userAgent = “Mozilla/5.0 (Windows NT 6.1; rv:6.0.2) Gecko/20100101 Firefox/6.0.2”;
var CPWcookies = new System.Net.CookieContainer();
HttpWebResponse webResponse = utilities.GetResponseWithMultipartPost(URL, userAgent, postParameters, CPWcookies);
var SNcookies = CPWcookies;
// Process response
StreamReader responseReader = new StreamReader(webResponse.GetResponseStream());
string htmlPage = responseReader.ReadToEnd();
webResponse.Close();
Thread.Sleep(Config.ThreadSleepTime.WebRequest);//give the server a break
// Pass the cookie to other url.
// Generate post objects
postParameters = new Dictionary<string, object>();
postParameters.Add(“Field0”, “1”);
postParameters.Add(“Field1”, “2”);
// Create request and receive response
webResponse = utilities.GetResponseWithMultipartPost(URL, userAgent, postParameters, SNcookies);
// Process response
responseReader = new StreamReader(webResponse.GetResponseStream());
htmlPage = responseReader.ReadToEnd();
webResponse.Close();
}
I added a new column to a table and my view is not showing this new column
use EXEC sp_refreshview viewname
Web Scraping in ASP.NET
i am taking the example of a page which pass parameters on post to itself.
first you need to identify what are the parameters when a page post.
you can identify with the firebug addone in firefox.
On the firebug toolbar, there is a net tab. click on the all tab under net tab. check the parameter for post.
the parameters will be just like Device=1&Manufacturer=Apple&Model=all
Now how to post parameters when working on screen scraping.
Following is the example of function, just pass the url and parameters.
public string GetResponseWithPost(string StrURL, string strPostData)
{
string strReturn = “”;
HttpWebRequest objRequest = null;
ASCIIEncoding objEncoding = new ASCIIEncoding();
Stream reqStream = null;
HttpWebResponse objResponse = null;
StreamReader objReader = null;
try
{
objRequest = (HttpWebRequest)WebRequest.Create(StrURL);
objRequest.Method = “POST”;
byte[] objBytes = objEncoding.GetBytes(strPostData);
objRequest.ContentLength = objBytes.Length;
objRequest.ContentType = “application/x-www-form-urlencoded”;
reqStream = objRequest.GetRequestStream();
reqStream.Write(objBytes, 0, objBytes.Length);
IAsyncResult ar = objRequest.BeginGetResponse(new AsyncCallback(GetScrapingResponse), objRequest);
//// Wait for request to complete
ar.AsyncWaitHandle.WaitOne(1000 * 60 * 3, true);
if (objRequest.HaveResponse == false)
{
throw new Exception(“No Response!!!”);
}
objResponse = (HttpWebResponse)objRequest.EndGetResponse(ar);
objReader = new StreamReader(objResponse.GetResponseStream());
strReturn = objReader.ReadToEnd();
}
catch (Exception exp)
{
throw exp;
}
finally
{
objRequest = null;
objEncoding = null;
reqStream = null;
if (objResponse != null)
objResponse.Close();
objResponse = null;
objReader = null;
}
return strReturn;
}
this function return the string containing html of page.
Now how to extract the valuable information from html.
There are many ways to extract the information.
1)Read the html and use the regular expression to find the information.
2)Read the html and find the tags in the html (the tags may be class name or id of div, depends which is near and unique on page)
Lets say, below is the information in html which i want to extract.
<div id=”manufactures”>
<ul>
<li>a</li>
<li>b</li>
<li>c</li>
</ul>
</div>
first i’ll search for <div id=”manufactures”> tag, then enable a extractdata flag to true and with the help of regular expression i will extract the words between the list
and after </ul> i will break the process of search.
Code:-
private static ArrayList ExtractBrands(string htmlPage, ref ArrayList BrandArray)
{
Regex searchExp = new Regex(@”<a href=\””(.+)\””>.+</a>”, RegexOptions.IgnoreCase);
string line = string.Empty;
string findWord = string.Empty;
bool extractData = false;
string[] strLines = Regex.Split(htmlPage, “\n”);
for (int loopI = 0; loopI < strLines.Length; loopI++)
{
line = strLines[loopI];
findWord = @”<div id=””manufacturers””>”;
if (line.Trim().Contains(findWord))
{
extractData = true;
continue;
}
if (extractData == true)
{
findWord = “</ul>”;
if (line.Trim().Contains(findWord))
{
extractData = false;
break;
}
MatchCollection MatchList = searchExp.Matches(line);
if (MatchList.Count > 0)
{
Match FirstMatch = MatchList[0];
Console.WriteLine(FirstMatch.Groups[1].Value);
BrandArray.Add(FirstMatch.Groups[1].Value);
}
}
}
return BrandArray;
}
What is SSRS and common problems faced in SSRS
i was worked on SSRS on last week and faced many issues while working on SSRS. i am sharing some of the issues with you.
What is SSRS
Reporting Services is a server-based reporting platform that provides comprehensive reporting functionality for a variety of data sources. Reporting Services includes a complete set of tools for you to create, manage, and deliver reports, and APIs that enable developers to integrate or extend data and report processing in custom applications. Reporting Services tools work within the Microsoft Visual Studio environment and are fully integrated with SQL Server tools and components.
With Reporting Services, you can create interactive, tabular, graphical, or free-form reports from relational, multidimensional, or XML-based data sources. You can publish reports, schedule report processing, or access reports on-demand. Reporting Services also enables you to create ad hoc reports based on predefined models, and to interactively explore data within the model. You can select from a variety of viewing formats, export reports to other applications, and subscribe to published reports. The reports that you create can be viewed over a Web-based connection or as part of a Microsoft Windows application or SharePoint site. Reporting Services provides the key to your business data.
Common Problems faced while working on SSRS.
1. How to Remove scrollbars in report.
set the properties SizeToReportContent= True AND AsynchRendering = False of reportview controls.
2. In IE7, if your report is cut from bottom, then you remove the line
<!DOCTYPE html PUBLIC “~//W3C//DTD XHTML 1.0 Transitional//EN” “http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd”>
from the webpage.
3. Sometime you can face a problem in report. it can show you
“Error encountered displaying report. The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.”
then you need to configure the SSL.
SSL is a little complex and there are a number of things to investigate.
When using SSL, using the incorrect URLs can result in failures like the one i listed above.
Check the SSL Certificate (steps for viewing certificates are listed below):
The value in Issued To is what you need to provide in the URL. If Issue To is “machine.domain.com” then typing http://localhost… in the browser will fail. Instead try https://<IssuedTo>…
Intended Purposes must include Server Authentication
Ensure the SSL Certificate is issued by a certificate authority recognized by your Domain Controller. Otherwise Report Manager will fail to connect to Report Server. Self signed certificates do not work.
In Reporting Services Configuration Manager:
Ensure a SSL URL is reserved and that a valid certificate is selected
Ensure the IP address selected for the certificate binding is correct
In rsreportserver.config
Set HostName property to the value of IssuedTo, or
Set ReportServerURL explicitly
To disable SSL by default set SecureConnectionLevel to 0
To see the certificates your using:
use mmc (Start –> run –> mmc –> enter)
Add the Certificates Add in (File –> Add/Remove Snap-in –> Add… –> Certificates)
Select Computer Account (Next –> Finish –> Close –> OK)
Under Console Root look at the “Personal” certificates. If you’re using a command line tool instead, the certificates are in the “MY” store.
Expand Certificates (Local Computer), Expand Personal, Click on Certificates
SSL can use any certificate in this store where the Intended Purposes list contains “Server Authentication”
4. If you face a error message like “HTTP Status 401 Unauthorized Error”
The Report server will expect authorized user to access the reports. This can be done by two ways.
1.Passing the Report Server User Credentials with the Reports.
2.Forms Authentication on Reporting Server.
We need to Create one sealed class to perform this action. This class need to be inherited from IReportServerCredential interface.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using Microsoft.Reporting.WebForms;
using System.Security.Principal;
[Serializable]
public sealed class ReportServerNetworkCredentials : IReportServerCredentials
{
#region IReportServerCredentials Members
/// <summary>
/// Provides forms authentication to be used to connect to the report server.
/// </summary>
/// <param name=”authCookie”>A Report Server authentication cookie.</param>
/// <param name=”userName”>The name of the user.</param>
/// <param name=”password”>The password of the user.</param>
/// <param name=”authority”>The authority to use when authenticating the user, such as a Microsoft Windows domain.</param>
/// <returns></returns>
public bool GetFormsCredentials(out System.Net.Cookie authCookie, out string userName,
out string password, out string authority)
{
authCookie = null;
userName = null;
password = null;
authority = null;
return false;
}
/// <summary>
/// Specifies the user to impersonate when connecting to a report server.
/// </summary>
/// <value></value>
/// <returns>A WindowsIdentity object representing the user to impersonate.</returns>
public WindowsIdentity ImpersonationUser
{
get
{
return null;
}
}
/// <summary>
/// Returns network credentials to be used for authentication with the report server.
/// </summary>
/// <value></value>
/// <returns>A NetworkCredentials object.</returns>
public System.Net.ICredentials NetworkCredentials
{
get
{
string userName = MobilePhoneXchange.WhiteLabel.Config.ServerUserName;
string domainName = MobilePhoneXchange.WhiteLabel.Config.Domain;
string password = MobilePhoneXchange.WhiteLabel.Config.ServerPassword;
return new System.Net.NetworkCredential(userName, password, domainName);
}
}
#endregion
}
protected void Page_Load(object sender, EventArgs e)
{
myReportViewer.ProcessingMode = Microsoft.Reporting.WebForms.ProcessingMode.Remote;
// Here we are going to pass the ReportServerCredentials to the Report Viewer.
myReportViewer.ServerReport.ReportServerCredentials = new ReportServerNetworkCredentials();
ReportServerLoaction = ConfigurationManager.AppSettings[“REPORT_SERVER_PATH”];
myReportViewer.ServerReport.ReportServerUrl = new Uri(“http://MyReportServer/Reports/”);
myReportViewer.ServerReport.ReportPath = “TempReports/MyFirstReport”;
myReportViewer.ShowParameterPrompts = false;
myReportViewer.ShowPrintButton = true;
Microsoft.Reporting.WebForms.ReportParameter[] reportParameterCollection = new Microsoft.Reporting.WebForms.ReportParameter[1];
reportParameterCollection[0] = new Microsoft.Reporting.WebForms.ReportParameter();
reportParameterCollection[0].Name = “ClientID”;
reportParameterCollection[0].Values.Add(“49020644-63AA-4D92-81A1-8F85D49ACF67”);
myReportViewer.ServerReport.SetParameters(reportParameterCollection);
myReportViewer.ServerReport.Refresh();
}
5. Sometime when you deploy to ReportServer, charts do not display or action on textbox is not working.
This issue can occur in IIS7
or sometimes When we migrate web applications from IIS 6 to IIS 7 or IIS 7.5, we will face some problems in http handlers, mappings etc.
For this you have to add handler.
Open Internet Information Services (IIS) Manager and select your Web
application.
Under IIS area, double-click on Handler Mappings icon.
At the Action pane on your right, click on Add Managed Handler.
At the Add Managed Handler dialog, enter the following:
Request path: Reserved.ReportViewerWebControl.axd
Type: Microsoft.Reporting.WebForms.HttpHandler
Name: Reserved-ReportViewerWebControl-axd
Click OK.
Which design pattern to use on a problem
The Factory pattern: You want to use it when creating (like) objects and don’t want to expose the recipient to managing a mess of dependencies. Think of it as having a class that maintains the necessary configuration info, so 10 distinct classes don’t need to get it as well and can focus on their business logic. You generally use factories with constructor dependencies, rather then setter injected ones.
The Strategy pattern: Its good when you are passing back an object with the SAME interface. The recipient doesn’t need to know how its being done to effect their business logic and its determined at run time where you get to choose what is most appropriate. Imagine writing a chess game, where the computer competitor was trying to tune itself to be challenging (but not impossible). As it analyzed your moves, it determined on the fly what expert personalities to use against you and how good of a player you are. This strategy changes repeatedly at run time, but maintains a constant interface. (Sorry, this is a bit of a tough one to find a great example for)
The Bridge pattern: This one is tough to get your head around. Its all about allowing the abstraction and implementation to vary independantly. I like the example of a household switch (the abstraction) and the light/fan/etc (the implementation). The switch can be implemented in many ways, from a two-position to a dimmer to a clapper. However, it does need to have a reference to the implementator to tell it what to do. However, that’s a standard interface (perhaps just a “On/Off” signal or a hard cut to the power). What it drives doesn’t matter and can change over time. Some day you decide to switch from a two-position to a voice activated one and later have it change from turning on the closet light to the indoor spa (you renovated). This is the same pattern, but in software.
The Adapter pattern: This one just acts as, well, an adapter. It converts from a U.S. power socket to a European one. It lets your VGA monitor plug into the DVI port. It lets two software systems (with similar intents) share information without having to know how the other handles data. For example, both may maintain student records, but have different views on what a “student” is. It can be as trivial as one maintaining rank as “freshman” and the other by year. Neither need to know about the existance of the other and screw with its business logic. An adapter is the only one knowing both and handles the conversion process.
Remove Recent Projects from Visual Studio 2008
Ever wanted to remove an item from the recent projects menu on the start page of Visual Studio?
The list is stored in the registry under:
HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\9.0\ProjectMRUList
Here, you will find a list like this:
File1 Reg_Expand_Sx Path
File2 Reg_Expand_Sx Path
File3 Reg_Expand_Sx Path
File4 Reg_Expand_Sx Path
You just need to delete the items you don’t want.
Note: If you delete item 2 you will need to rename item 3 and 4 so there are no gaps in the naming. (3 becomes 2, 4 becomes 3).
Another way is just to wait until you have opened more projects. Also, if you delete the project solution and try and open it, Visual Studio will display a dialog asking if you want to remove it from the list.
Code Junction 