... . send an HTTP redirect, use redirect:http://www.hsc.fr/ The example provides: if SubWeb finds the word 'redirect' in the URL it will redirect the client on http://www.hsc.fr/ Traffic visualization --------------------- It is possible to visualize all the traffic between the customers and the servers. There are several options of visualization (only the headers, to display binary pages in Hexa, to display only the requests or only the answers...) Exemple de visualisation : -------------------------- Connected to the Internet, run : ./subweb Via a proxy, run : ./subweb 8080 midproxy corporate_proxy 3128 Configure the 'proxy' in your browser on localhost:3128 Go to: http://www.google.com/ Results: -------- [Fri Aug 10 16:32:59 2001: connection from localhost [ 127.0.0.1 ] at port 45929] GET http://www.google.com/ HTTP/1.0 User-Agent: Mozilla/5.0 (Linux 2.4.4 i686; U) Opera 5.0 [en] Host: www.google.com Accept: text/html, image/png, image/jpeg, image/gif, image/x-xbitmap, */* Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0 Coockie: NO-COOKIE-PLEASE:NO-COOKIE-PLEASE:NO-COOKIE-PLEASE:NO-COOKIE-PLEASE Cookie2: $Version="1" Pragma: no-cache Cache-Control: no-cache -- HTTP/1.0 200 OK Date: Fri, 10 Aug 2001 14:36:13 GMT Server: GWS/1.11 Set-Cookie: PREF=ID=1253841f47eb7ec4:TM=997454173:LM=997454173; \ domain=.google.com; path=/; expires=Sun, 17-Jan-2038 19:14:07 GMT Content-Type: text/html Content-Length: 2184 Cache-Control: private X-Cache: MISS from carbone.hsc.fr Proxy-Connection: close Filtering --------- In the three modes it is possible to apply filtering at all the levels, ie. in the URL, the headers and the body of the pages, in the requests and the answers. Filtering is active if the variable $static_filters is not null. Another type of filtering, named dynamic, is activable by adding the string subweb=on in the URL. These dynamic filters are interesting, for example, to change fields like a cookie or a session_id in a session after passing the authentification. Static and dynamic filtering is configured in the functions: FilterIN, FilterOUT, DynamicFilterIN and DynamicFilterOUT. FilterIN(), DynamicFilterIN() ^ | | v /--------\ Request /--------\ /--------\ | |------------->| |-------------->| | | Client | | SubWeb | | Server | | |<-------------| |<--------------| | \--------/ \--------/ Answer \--------/ ^ | | v FilterOUT(), DynamicFilterOUT() Example of filtering function: ------------------------------ sub FilterIN { my $request = shift; ## Put your filters here - from client to server ## ## Change the user-agent $request =~ s/^User-Agent:\s+\S+$/User-Agent: SubWeb/gm; # Trivial anti-ads $request =~ s/\.doubleclick\.net/xxx/g; # Cookie stuff # Don't send coockies # $request =~ s/^Cookie:\s+.*\r\n//gm; # activism ;) $request=~s/Cookie:\s+.*\r\n/sprintf("Cookie: %s\r\n","NO-COOKIE:"x5)/egm; return $request; } The filtering functions are thus implemented to make it possible to the user to develop his own filters without any restriction. Modification of requests GET in POST ------------------------------------- SubWeb makes it possible to transform all requests GET into POST automatically and recomputes the Content-length. Force or change authentification HTTP ------------------------------------- It is possible with $force_auth and $http_auth to force or replace HTTP authentification thus in a transparent way several users can use the proxy which authenticates all the requests on another proxy or on a Web server. Test intrusion detection systems (IDS) -------------------------------------- SubWeb has six modes making it possible to make illegible the URL. With : $anti_ids_modes = '1523' the request GET /fr.yahoo.com/ becomes: GET http://fr.yahoo.com/%2e/%76%7a%73%78%74%69%66%67%64%68%78%6e %73%6a%6f%7a%67%77%68%78%6e%6b%79%7a%70%6a%68%66%6b%65%76%77%65%61 %63%65%68%6e%63%78%75%73%72%68%66%72%70%75%79%66%64%79%70%75%62%65 %77%66%6f%6c%63/%2e/%2e%2e/%2e/ Encryption of the fields hidden in reverse proxy mode ----------------------------------------------------- An experimental functionality was added to the reverse relay mode. It makes it possible to cipher the contents of the hidden fields sent by the server to the various customers and to decipher them when the customers send them back to the server in requests GET or POST requests. This mechanism forbidden the users to modify the value of the hidden fields, which makes it possible to protect them and use them for example to manage the order of the requests required by a customer. With such a mechanism it can be very difficult to crack applications based on HTTP. It is activated when the variable $cypher_hidden_field is nonnull and when the mode is reverse proxy. The encryption algorithm used is Blowfish provided in the Crypt::Blowfish module. The encryption key can be provided in the variable $passphrase_hidden and must contains 32 hexa char. If it is not provided SubWeb will use a pseudo-random one. Syntax ====== By defect, SubWeb listens on the port 8080/tcp and is in 'midproxy' mode on the address localhost and the port 3128/tcp. ie. that SubWeb awaits requests of your navigator on port 8080 and tries to send them on your local squid relay. To change the port on which SubWeb is put in listening it is necessary to launch it with a first parameter, example for port 8081: % ./subweb 8081 To change the mode it is necessary to add a second parameter: % ./subweb 8081 [ proxy | midproxi | rproxy ] By defect the server and the distant port are localhost and 3128/tcp. To change this server and this port 2 other parameters should be added: % ./subweb 8081 rproxy webserver 80 Or: % ./subweb 8080 proxy Or: % ./subweb 8081 midproxy httproxy 8080 Options ======= my $signature = 1; # to activate the SubWeb signature my $convert_get_to_post = 1; # to activate the transformation of the GET into POST my $force_auth = 1; # to force http authenticication to base64($http_auth) my $http_auth = "aubert:kotao"; # to give an http authentication login:passwd my $anti_ids_modes = '1523'; # to activate the anti-ids modes my $accept_gzipped_pages = 1; # allows to authorize the server to compress the answers my $static_filters = 1; # to activate static filtering my $show_only_url = 1; # to show URL only in requests my $dump_text_body = 1; # to show the contain of text pages my $dump_binary_body = 1; # to show the contain of binary pages (in hexa) my $showIN = 1; # to show requests my $showOUT = 1; # to show answers my $showUNFILTERED = 1; # to show requests and answers before filtering my $showFILTERED = 1; # to show requests and answers after filtering PS: SubWeb is not underground ;) --------------------------------

SubWeb v1.0 Stephane Aubert kotao HSC security research labs Hervé Schauer Consultants Download: http://www.hsc-labs.com/tools/subweb/ Description =========== SubWeb is a proxy (and also a reverse proxy). It allows to work on HTTP flows in the line of HTTPush, RFProxy or Achilles. It becomes possible with SubWeb to handle and visualize on the fly the HTTP requests, the headers and/or HTML pages. Main goal of SubWeb is to contribute to the tests of network applications based on HTTP. HTTPS is not directly managed in SubWeb, it is necessary, in order to test a SSL server, to use the stunnel program, for example. SubWeb has 3 operating modes: * proxy (classical HTTP proxy) /--------\ Request /--------\ /--------\ | Client |------------->| SubWeb |-------------->| Server | | |<-------------| proxy |<--------------| | \--------/ \--------/ Answer \--------/ * midproxy (HTTP proxy which requires the pages of another proxy) /--------\ Req. /--------\ /--------\ /--------\ | Client |------>| SubWeb |---->| HTTP |------>| Server | | |<------|midproxy|<----| proxy |<------| | \--------/ \--------/ \--------/ Ans.\--------/ * rproxy (reverse proxy, SubWeb mimic an HTTP server) /--------\ /--------\ Request /--------\ | Server |<-------------| SubWeb |<--------------| Client | | |------------->| rproxy |-------------->| | \--------/ Answer \--------/ \--------/ Virtual Web ----------- Another functionality, named virtual Web, allow SubWeb to answer certain requests (depending on keywords contained in these requests) without requiering anything to the server. /--------\ Request /--------\ /--------\ | Client |------------->| SubWeb | | Server | | |<-------------| proxy | | | \--------/ Answer \--------/ \--------/ VirtualWeb can be configured with $virtual_web to activate this "module" and the hash-table %vweb_config to provide actions according to key words. The actions available for VirtualWeb are . send a file from disk, use file:/tmp/foo . send a file from string, use html: ... . send an HTTP redirect, use redirect:http://www.hsc.fr/ The example provides: if SubWeb finds the word 'redirect' in the URL it will redirect the client on http://www.hsc.fr/ Traffic visualization --------------------- It is possible to visualize all the traffic between the customers and the servers. There are several options of visualization (only the headers, to display binary pages in Hexa, to display only the requests or only the answers...) Exemple de visualisation : -------------------------- Connected to the Internet, run : ./subweb Via a proxy, run : ./subweb 8080 midproxy corporate_proxy 3128 Configure the 'proxy' in your browser on localhost:3128 Go to: http://www.google.com/ Results: -------- [Fri Aug 10 16:32:59 2001: connection from localhost [ 127.0.0.1 ] at port 45929] GET http://www.google.com/ HTTP/1.0 User-Agent: Mozilla/5.0 (Linux 2.4.4 i686; U) Opera 5.0 [en] Host: www.google.com Accept: text/html, image/png, image/jpeg, image/gif, image/x-xbitmap, */* Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0 Coockie: NO-COOKIE-PLEASE:NO-COOKIE-PLEASE:NO-COOKIE-PLEASE:NO-COOKIE-PLEASE Cookie2: $Version="1" Pragma: no-cache Cache-Control: no-cache -- HTTP/1.0 200 OK Date: Fri, 10 Aug 2001 14:36:13 GMT Server: GWS/1.11 Set-Cookie: PREF=ID=1253841f47eb7ec4:TM=997454173:LM=997454173; \ domain=.google.com; path=/; expires=Sun, 17-Jan-2038 19:14:07 GMT Content-Type: text/html Content-Length: 2184 Cache-Control: private X-Cache: MISS from carbone.hsc.fr Proxy-Connection: close Filtering --------- In the three modes it is possible to apply filtering at all the levels, ie. in the URL, the headers and the body of the pages, in the requests and the answers. Filtering is active if the variable $static_filters is not null. Another type of filtering, named dynamic, is activable by adding the string subweb=on in the URL. These dynamic filters are interesting, for example, to change fields like a cookie or a session_id in a session after passing the authentification. Static and dynamic filtering is configured in the functions: FilterIN, FilterOUT, DynamicFilterIN and DynamicFilterOUT. FilterIN(), DynamicFilterIN() ^ | | v /--------\ Request /--------\ /--------\ | |------------->| |-------------->| | | Client | | SubWeb | | Server | | |<-------------| |<--------------| | \--------/ \--------/ Answer \--------/ ^ | | v FilterOUT(), DynamicFilterOUT() Example of filtering function: ------------------------------ sub FilterIN { my $request = shift; ## Put your filters here - from client to server ## ## Change the user-agent $request =~ s/^User-Agent:\s+\S+$/User-Agent: SubWeb/gm; # Trivial anti-ads $request =~ s/\.doubleclick\.net/xxx/g; # Cookie stuff # Don't send coockies # $request =~ s/^Cookie:\s+.*\r\n//gm; # activism ;) $request=~s/Cookie:\s+.*\r\n/sprintf("Cookie: %s\r\n","NO-COOKIE:"x5)/egm; return $request; } The filtering functions are thus implemented to make it possible to the user to develop his own filters without any restriction. Modification of requests GET in POST ------------------------------------- SubWeb makes it possible to transform all requests GET into POST automatically and recomputes the Content-length. Force or change authentification HTTP ------------------------------------- It is possible with $force_auth and $http_auth to force or replace HTTP authentification thus in a transparent way several users can use the proxy which authenticates all the requests on another proxy or on a Web server. Test intrusion detection systems (IDS) -------------------------------------- SubWeb has six modes making it possible to make illegible the URL. With : $anti_ids_modes = '1523' the request GET /fr.yahoo.com/ becomes: GET http://fr.yahoo.com/%2e/%76%7a%73%78%74%69%66%67%64%68%78%6e %73%6a%6f%7a%67%77%68%78%6e%6b%79%7a%70%6a%68%66%6b%65%76%77%65%61 %63%65%68%6e%63%78%75%73%72%68%66%72%70%75%79%66%64%79%70%75%62%65 %77%66%6f%6c%63/%2e/%2e%2e/%2e/ Encryption of the fields hidden in reverse proxy mode ----------------------------------------------------- An experimental functionality was added to the reverse relay mode. It makes it possible to cipher the contents of the hidden fields sent by the server to the various customers and to decipher them when the customers send them back to the server in requests GET or POST requests. This mechanism forbidden the users to modify the value of the hidden fields, which makes it possible to protect them and use them for example to manage the order of the requests required by a customer. With such a mechanism it can be very difficult to crack applications based on HTTP. It is activated when the variable $cypher_hidden_field is nonnull and when the mode is reverse proxy. The encryption algorithm used is Blowfish provided in the Crypt::Blowfish module. The encryption key can be provided in the variable $passphrase_hidden and must contains 32 hexa char. If it is not provided SubWeb will use a pseudo-random one. Syntax ====== By defect, SubWeb listens on the port 8080/tcp and is in 'midproxy' mode on the address localhost and the port 3128/tcp. ie. that SubWeb awaits requests of your navigator on port 8080 and tries to send them on your local squid relay. To change the port on which SubWeb is put in listening it is necessary to launch it with a first parameter, example for port 8081: % ./subweb 8081 To change the mode it is necessary to add a second parameter: % ./subweb 8081 [ proxy | midproxi | rproxy ] By defect the server and the distant port are localhost and 3128/tcp. To change this server and this port 2 other parameters should be added: % ./subweb 8081 rproxy webserver 80 Or: % ./subweb 8080 proxy Or: % ./subweb 8081 midproxy httproxy 8080 Options ======= my $signature = 1; # to activate the SubWeb signature my $convert_get_to_post = 1; # to activate the transformation of the GET into POST my $force_auth = 1; # to force http authenticication to base64($http_auth) my $http_auth = "aubert:kotao"; # to give an http authentication login:passwd my $anti_ids_modes = '1523'; # to activate the anti-ids modes my $accept_gzipped_pages = 1; # allows to authorize the server to compress the answers my $static_filters = 1; # to activate static filtering my $show_only_url = 1; # to show URL only in requests my $dump_text_body = 1; # to show the contain of text pages my $dump_binary_body = 1; # to show the contain of binary pages (in hexa) my $showIN = 1; # to show requests my $showOUT = 1; # to show answers my $showUNFILTERED = 1; # to show requests and answers before filtering my $showFILTERED = 1; # to show requests and answers after filtering PS: SubWeb is not underground ;) --------------------------------