DataDirect HTTP Functions

This section describes the DataDirect XQuery HTTP functions you can use to implement an XQuery Web service client, including how to take advantage of some of the low-level connection functionality provided by the HTTP functions that is not available with ddtek:wscall.

Function Overview

DataDirect XQuery provides function declarations to support the following HTTP methods on HTTP 1.0 and 1.1:

DELETE (ddtek:http-delete) – requests that the server delete the specified resource

GET (ddtek:http-get) – retrieves from the server the resource specified in the URI

HEAD (ddtek:http-head) – requests that the server return only header information for the specified resource

OPTIONS (ddtek:http-options) – a request for available communication options

POST (ddtek:http-post) – typically used to submit a resource to the server for additional processing

PUT (ddtek:http-put) – typically used to insert or replace a server resource

TRACE (ddtek:http-trace) – requests that the server echo back the request it received

Example – ddtek:http-get

This example shows a simple use of ddtek:http-get. Here, the XQuery requests the resource hello.txt on http://examples.xquery.com:

The result is returned in a <response> element, with the data in the <response-body>:

<response http-version="HTTP/1.1" status-code="200" reason="OK"> 
	<response-header> 
		<header name="ETag" value='W/"5-1243350649093"'/> 
		<header name="Date" value="Wed, 24 Jun 2009 20:40:25 GMT"/> 
		<header name="Content-Length" value="5"/> 
		<header name="Last-Modified" value="Tue, 26 May 2009 15:10:49 GMT"/> 
		<header name="Content-Type" value="text/plain"/> 
		<header name="Server" value="Apache-Coyote/1.1"/> 
	</response-header> 
	<response-body>hello</response-body> 
</response>

Connection Authentication

Before a Web service client application can be run, the client must connect to the origin server. Each connection is authenticated by the Web server. DataDirect XQuery supports these authentication methods for HTTP functions:

These authentication methods are separate from any that might be used by applications to access relational database tables using the collection function. See Securing Data Source Connections to learn about establishing secure connections using NTLM and Kerberos.

The Authentication Process

The type of authentication performed is established by the origin or proxy server. Consider the following example of an invocation of the ddtek:http-get function:

ddtek:http-get(
 'http://examples.xquery.com/secure/members/books.xml',
 <request username="ddtek" password="ddtek"/>
)

Note that the invocation does not specify which authentication method to use, even though the <request> element specifies username= and password= attributes, which suggests an awareness of the need to provide the server with this information. (See Specifying HTTP Client-Server Options to learn more about the <request> element.)

Rather, the required authentication method is provided by the server through a series of challenges to and responses from the XQuery Web service client, as shown in the following simplified exchange:

The ddtek:http-get function attempts to connect to a resource on http://examples.xquery.com:

The server denies the request. Included in the response is the authentication method it requires:

				HTTP/1.1 401 Unauthorized 
				WWW-Authenticate: Digest realm="Authentication Test", 
				qop="auth", nonce="4b3ab6dd951a22816cd32c763ea415e6", 
				opaque="d14db1a1aba28fd461ce63a8dac78069"

The request is sent again, this time including the information required by the server – the authorization method, the realm, and a password encrypted using that authentication method:

				GET http://examples.xquery.com/secure/members/books.xml HTTP/1.1 
				Authorization: Digest username="ddtek", 
				realm="Authentication Test", 
				nonce="4b3ab6dd951a22816cd32c763ea415e6", 
				uri="/secure/members/books.xml"

Once the authorization succeeds, the server responds with an OK and the XQuery can be executed.

This entire exchange, after the initial request by the ddtek:http-get, takes place without intervention from the user.

Supported Encryptions

DataDirect XQuery supports encryption with the HTTPS (HTTP over SSL) protocol. No additional parameter is required to use SSL (Secure Sockets Layer) – it is indicated by the URI scheme HTTPS.

Consider the following example, which connects to the Verisign^® secure Web site:

This query generates the following result (the response body has been omitted for formatting considerations):

<response http-version="HTTP/1.1" status-code="200" reason="OK">
    <response-header>
        <header name="Content-type" value="text/html"/>
        <header name="Date" value="Mon, 01 Jun 2009 15:39:19 GMT"/>
        <header name="Set-Cookie" value="v1st=4A23F627FE4EFAD6; path=/;
          expires=Wed, 19 Feb 2020 14:28:00 GMT; domain=.verisign.com"/>
        <header name="Connection" value="close"/>
        <header name="Server" value="Netscape-Enterprise/4.1"/>
    </response-header>
    <response-body>...</response-body>
</response>

Managing Connections and Sockets

Each DataDirect XQuery HTTP function is associated with an underlying connection manager. Each connection manager can keep one, and only one, connection open.

When a sequence of calls is directed to the same host, the connection is reused. For example, the following XQuery code executes a sequence of HTTP PUT functions in order to upload a set of XML files from a local directory (c:/docs) to a WebDAV server (http://examples.xquery.com/upload/):

declare variable $host := "http://examples.xquery.com/upload/"; 
 
<result>{ 
for $d in collection("file:///c:/docs?select=*.xml")  
let $filname := tokenize(document-uri($d), '/')[last()] 
return  
  ddtek:http-put(concat($host, $filname),$d) 
}</result>

Contrast the previous example with this one, which uses two connections because it is uploading the same document (books.xml) to two different servers (http://localhost and http://remotehost):

<result>{ 
ddtek:http-put( 
 "http://localhost/books.xml", doc("file:///c:/books.xml")),
ddtek:http-put( 
 "http://remotehost/books.xml", doc("file:///c:/books.xml")) 
</result>

Settings for Connections and Sockets

DataDirect XQuery provides numerous settings that allow you more direct control over the connections and sockets associated with your XQuery Web service client. Examples of connection and socket settings you can specify include connection and socket timeout values and the number of retries.

You specify connection and socket settings as attributes of the <request> element, which can be included as an additional parameter of the DataDirect XQuery HTTP function you are invoking, as shown in the following example:

Here, the <request> element defined as part of the ddtek:http-get function specifies that Base64 should be used for the response encoding, and that the HTTP call should be tried a maximum of four times.

The <request> element attributes you can use to manage connection and sockes settings are:

connection-timeout

password

protocol-head-body-timeout

protocol-reject-head-body

proxy-host

proxy-password

proxy-port

proxy-username

retries

socket-linger

socket-receivebuffer

socket-sendbuffer

socket-timeout

username

Data Streaming

All DataDirect XQuery HTTP functions are designed to support Streaming XML, a DataDirect technique that processes data sequentially, allowing efficient transmission of data input and output during query execution.

There might be times, however, when you want to turn off streaming in order to preserve resources on the origin server. When a query invokes a large number of HTTP calls, for example, each call opens a dedicated connection which can ultimately lead to degraded performance.

Disabling Streaming XML

Set ddtek:xml-streaming="no" in the query prolog. See Using Option Declarations and Extension Expressions for more information on this topic.

Set the "streaming" attribute in the <request> element to no. See Specifying HTTP Client-Server Options to learn more about the <request> element.

Parameter Values and Streaming XML

In contrast to data, which might or might not be streamed, parameter values are always fully materialized before the calling function is invoked. If a query function uses a document as its input, for example, the entire document is loaded in memory before the function is invoked.

Response Encoding

DataDirect XQuery automatically encodes the raw data streams returned by ddtek:http-get and ddtek:http-post functions. The encoded service response is placed in an element referred to as the payload for consumption by the XQuery code.

<response http-version="HTTP/1.1" status-code="200" reason="OK"> 
	<response-header> 
		<header name="ETag" value='W/"5-1243350649093"'/> 
		<header name="Date" value="Wed, 24 Jun 2009 20:40:25 GMT"/> 
		<header name="Content-Length" value="5"/> 
		<header name="Last-Modified" value="Tue, 26 May 2009 15:10:49 GMT"/> 
		<header name="Content-Type" value="text/plain"/> 
		<header name="Server" value="Apache-Coyote/1.1"/> 
	</response-header> 
	<response-body>hello</response-body> 
</response>

The response-header provides information about the resource being queried (in this case, a file called hello.txt), including its Content-Type (which is "text/plain").

The response-body ("hello") represents the data returned by the service, which has been encoded by the DataDirect XQuery Web service client.

Encoding Rules

The method used to encode the service response depends on a number of factors, the most important of which is the Content-Type in the response header – if DataDirect recognizes the mime type value in the Content-Type provided in the response header, it provides a suitable encoding method. Otherwise text encoding is used.

If the origin server does not return a Content-Type in the response header, or if the mime type is not recognized, the response value is encoded as text.

If the origin server returns a Content-Type in the response header that contains a mime type recognized by DataDirect XQuery, one of the following encoding method is used, as appropriate: text, xml, or base64. (See Recognized Mime Types for a complete list of types and associated encodings.)

If the Content-Type is a text mime type (text/html, for example) and it includes a charset parameter, as shown in the following example, the charset is used to interpret the bytes in the response:

Content-Type: text/html; charset=utf-8

If the charset parameter is missing, DataDirect XQuery uses ISO-8859-1.

Overriding the Default Encoding

You can override the default encodings used by DataDirect XQuery. You might want to do this, for example, when the Content-Type is not specified in the response header but you know what it is.

Consider the following example. Here, we are using the HTTP GET function to query a file:

The .psd extension indicates that it is a graphics file, which has a mime type of image/x-photoshop. Because this mime type is not recognized by DataDirect XQuery, the default encoding method – text – is used. This ultimately causes the query to fail as the text encoding results in the creation of characters like , which are not valid XML.

To avoid this problem, we can override the default encoding using the response-data-type attribute of the <request> element, as shown here:

With the response encoding set to a binary type, the query can now be executed successfully:

<response http-version="HTTP/1.1" status-code="200" reason="OK"> 
	<response-header> 
		<header name="ETag" value='W/"18016-1243352017406"'/> 
		<header name="Date" value="Tue, 23 Jun 2009 21:02:19 GMT"/> 
		<header name="Content-Length" value="18016"/> 
		<header name="Last-Modified" value="Tue, 26 May 2009 15:33:37 GMT"/> 
		<header name="Content-Type" value="image/x-photoshop"/> 
		<header name="Server" value="Apache-Coyote/1.1"/> 
	</response-header> 
	<response-body>OEJQUw...</response-body> 
</response>

Note that the response body (which has been abbreviated here for formatting considerations) is the base64 encoding of the binary file format.

Recognized Mime Types

The following table summarizes the mime types recognized by DataDirect XQuery and the associated method used encode the response.

Table 10-1. Recognized Mime Types and Associated Encodings
Mime Type	Encoding
application/atom+xml	xml
application/base64	base64
application/mathml+xml	xml
application/rss+xml	xml
application/xhtml+xml	xml
application/xml	xml
application/xslt+xml	xml
application/zip	base64
image/svg+xml	xml
image/bmp	base64
image/gif	base64
image/jpeg	base64
image/png	base64
image/tiff	base64
text/html	text
text/plain	text
text/richtext	text
text/xml	xml
x-gzip	base64
x-compress	base64

Managing Cookies

Cookies are messages exchanged between a Web server and client that are used to manage HTTP state. Cookies commonly store information about visits to a Web site such as a user name, password, the last time a site was visited, and so on. There are several standards for cookies. Some, like RFC2109 and RFC2965, were drafted by the W3C; others, like the Netscape cookie specification, are vendor-specific.

By default, DataDirect XQuery supports the RFC2109 standard, but you can use the cookie-policy attribute of the <request> element to specify how you want your XQuery application to manage cookies. Available cookie management policies are summarized in the following table.

Table 10-2. <request> Element cookie-policy Parameters
Value	Description
RFC2109	RFC2109 was the first W3C cookies specification. Although widely used, RFC2109 is sometimes too strict for servers supporting other specifications. If you encounter compatibility issues with RFC2109, consider using RFC2965.
RFC2965	RFC2965 is the second version of the W3C RFC2109 cookies specification, intended to loosen some of the restrictions that made RFC2109 incompatible with some servers. Some of the key difference of RFC2965 include: RFC2965 cookies are port-sensitive Servers that send RFC2965 cookies will use both Set-Cookie2 and Set-Cookie headers; other cookie implementations use only Set-Cookie
netscape	The Netscape cookie specification formed the basis for RFC2109. However, differences between the two might require the use of Netscape specification on some servers.
ignore_cookies	This setting ignores all cookies – cookies are neither sent nor accepted when the cookie-policy attribute is set to ignore_cookies.

Specifying HTTP Client-Server Options

Every DataDirect XQuery HTTP function allows you to specify client-server options using a <request> element, as shown in the following example:

Here, the response-data-type attribute is being used to specify the encoding method to be used for the graphics file dis-logo.psd, which is being queried using the ddtek:http-get function.

Other examples of <request> element attributes include connection and socket timeout values, the number of retries, and HTTP version.