Feb 27 2008
Bi-Directional HTTP Transformation
The ability to transform and inspect HTTP data as it flows in and out of a web application has many practical uses (both inside and outside of security). On IIS, this capability was historically restricted to ISAPI filters. Http Modules written in ASP.NET have always allowed processing of requests and responses to and from an ASP.NET application, but with the advent of IIS7 and the integrated ASP.NET pipeline, Http Modules written in ASP.NET now have access to virtually all stages of request processing (including those not handled by ASP.NET).
Transformer.NET is an Http Module designed for on-the-fly inbound and outbound url rewriting. Apache’s mod_rewrite, used to manipulate inbound request urls, is arguably one of the most popular Apache modules around. While there have been several ports of mod_rewrite to IIS (with implementations ranging from Http Modules to ISAPIs), they all share one shortcoming in common with their Apache predecessor: They only rewrite requests and not urls within outbound responses (such as links that are generated within an HTML page).
This has long been a pet peeve of mine. If you want to use mod_rewrite, you typically need to update the underlying website source code so that the hyperlinks within the application point to the “rewritten” urls. This can be a major effort and inconvenience if the site is already written, and even worse, it may not be possible for 3rd party or COTS web applications.
The initial beta release of Transformer.NET differs from previous rewrite modules because it supports bi-directional (inbound and outbound) url rewriting. Bi-directional rewriting eliminates the need to modify the underlying website code, which is great for legacy or third party web sites and applications. In addition to the ability to parse response content (such as HTML), Transformer.NET also includes the following two key internal mechanisms:
|
Normalization Engine
Internal Url Cache |
The bi-directional rewriting capability of Transformer.NET was really an initial “proof-of-concept” for us to start building bi-directional HTTP inspection and transformation solutions to solve some very interesting web application security problems.
Unlike Apache’s mod_rewrite, Transformer.NET does not implement conditional rewrites (ala mod_rewrite’s RewriteCond) so it is not intended to be a total port. The current beta version can be downloaded from our tools page. Transformer works on IIS6 (limited to ASP.NET applications) and with any site running on IIS7. A detailed user guide is included with the download.
Inspect what? OWASP T10-2007 A1 and A2? Is this like a WAF? Can it be as granular as Core Grasp?
What is the the goal and what is this attempting to prevent?
There might be quite a lot you can do depending on the support in Transformer.NET. Check out some of the ideas from http://wsgw.sf.net
The current beta version is not meant as a security tool, but does perform inbound/outbound re-writes of URLS via mod_rewrite style rules (Regular Expressions and Rewrite Maps). Theoretically it can be used to deny certain URL patterns via the “Forbidden” action on a rewrite rule, but security was not the main intent. This was more of an initial proof of concept that was developed to make website url’s “Search Engine Friendly” without having to modify the site code.
Now that the framework is in place, we are going to add some features to do application security stuff like input filtering and URL/Form tamper protection. The concept is similar to mod_security, but for IIS instead of Apache (no separate hardware required). You can expect an updated blog post once these features are added.
Is this for IIS6 or IIS7??
What are the installation pre-requisites?
If this is for IIS6, and if it can be used on shared ASP.NET hosting, would you please provide specific, detailed nformation about how to use this to create DotNetNuke friendly URLs??
Many people would find this very useful, since DNN’s built-in friendly URLs has a lot of issues with spaces and special characters.
There are a couple other ones out there for DNN, but they require being able to change IIS configs on the webserver, which most people with shared hosting can’t do.
Thank you, Tom
Hi Tom –
For an ASP.NET application like DotNetNuke (DNN), Transformer.NET will work seamlessly on both IIS6 & IIS7. The only prerequisite would be that you will need to make a couple of entries into your web.config file (I presume you should have the ability to do this in the shared environment).
Here are a couple quick shortcuts to get it working (full documentation / user guide is included in the Zip download):
1 - Add the following entry under the
<configSections>area of your web.config:<section name="transformerConfiguration" type="Transformer.Net.Config.Options" />2 - Add the following section as a child node of the
<configuration>element of the web.config (you should be able to add this section directly after the closing</configSections>tag):<transformerConfiguration logToBrowser="true" logLevel="Low"></transformerConfiguration>
NOTE: You will need to add some Rewrite Rules within this section after you get it working based on your specific re-writing objectives
3 - Add the following entry under the
<httpModules>area of the web.config (under <system.web> for IIS6). Note that to avoid conflicting with other DotNetNuke HttpModules, you should make this the *first* module in the list.<add name="Transformer.Net" type="Transformer.Net.CoreModule, Transformer.Net"/>4 - Test the installation by making a request to one of the DNN pages. You should see some Transformer.NET log entries appended to the bottom of the page.
5 - Generate Rewrite Rules. There are a number of re-write rules you can create depending on your intention here. These rules are defined within the
<transformerConfiguration>web.config section we created in step 2. I would suggest consulting the User Guide (included in the Zip download) for specifics on how to do this since it will really vary depending on your objective. If you are familiar with the general syntax used by mod_rewrite, then you should be in good shape.Hopefully this helps out. Once the rules are working as intended, you should disable all logging by setting logToBrowser=”false” logLevel=”Off” within the root “transformerConfiguration” element. If you let me know your specific objectives (i.e. what you are hoping to accomplish with the re-write rules) I would be happy to help you come up with some specific rules if you are having problems. Feel free to post back here or send an e-mail to our tools e-mail alias.
Hello, thank you for your reply.
I will try it…but some examples and information etc. particularly for the DNN stuff would be very helpful.
I’ll email back after I have tried it out a little.
Thank you, Tom
Wow. This looks like exactly what I have been getting ready to build from scratch for a new client who has hired us for SEO but doesnt want to leave their current CMS. My only question is if there is an open source version available or a provider model of some sort for the rules and maps. I would prefer to use a db provider for these. In this project we would need to create thousands of map entries. I would be concerned about performance w/ that many rules in a file, as well they regularly add/change/delete categories, which would then require constant maintenance. Anyways, I read through your docs and it seems like Transformer.NET fits my needs to a tee except for this.
Thanks a lot, Duane
Duane: Unfortunately the code has not been open sourced since a good portion of it is currently being used for another yet-to-be-released GDS project. However, feel free to contact me directly via our tools e-mail alias (tools [at] gdssecurity.com) and perhaps we can come up with some ideas on how to make it work well for your specific scenario.
You write that it is a limitation to IIS6 that outbound ASP content cannot be rewritten. While looking for a good SEO solution we have found and successfully tested LinkFreeze. This tool IS able to rewrite the whole outbound ASP stuff - unfortunately it is very limited in it’s capabilites. No way to achieve this for you tool?
Marc: The limitation relates to HttpModules on IIS6, not IIS6 in general. ISAPIs have long had the ability to modify all parts of a request and response on IIS6. I would guess that the component you are referring to is an ISAPI and not an HttpModule.
This looks like what I’m looking for…
I have a situation where a sw application company sold us a web based app that has a home page URL of http://www.company.com/dir1/file.dll/home. (sheesh…)
They won’t modify the configuration because they consider it “customization” and will charge extra do it.
I need to be able to rewrite browser requests and responses to “hide”
the /dir1/file.dll/ part of the URL, so the new home page would be http://www.company.com/home and all browser responses would not contain the /dir1/file.dll/ part of the URL.
I already do this for external(internet) browser users with a reverse proxy server but the proxy is bypassed for internal (intranet) users.
1. Will this work for this application?
2. Will it work wil SSL?
3. Do you have a rule to share that I could use for testing?
Thanks
hitec: This module should do the trick. SSL has no impact on the module since it runs within IIS. The only caveat would depend on whether you are running on IIS6 or IIS7. If you are running IIS7, two rules should fix all of your URLs without any change to the underlying site.
Based on your post, I would guess they should look something like the following:
<add ruleid="in-1" direction="in" from="^/(.*)\.htm$" to="/dir1/file.dll/$1" />
<add ruleid="out-1" direction="out" from="^/dir1/file\.dll/(.*)" to="/$1.htm" />
This, of course, is just a guess based on the limited amount of information you provided. You’ll notice above that I added the “.htm” file extension to the re-written URLs. This is because without the “dir1/file.dll” portion of the URL, there is nothing left to uniquely identify the requests that need to be routed to the DLL vs any other request (like one for a .gif image). Assuming there are no .htm files on your server, this rule will properly route all .htm requests to the DLL.
If you are running on IIS6, you might be out of luck with the outbound URL re-writing. HttpModules on IIS6 only seem to have access to the response stream for managed handlers (aka .NET) and not other handlers. In either case, I hope this helps.
Hi Brian
I’m using IIS 6.0 with w2003 R2 SP2
I would of liked to try this module.
Thanks anyway