Wednesday, July 6, 2011

Transfer large messages in WCF - Part 3

This in continuation with Large Message Transfer in WCF Part 1 and Part 2.

In last article by using Buffered transfer mode and MTOM, we upload a file of 88 mb, we observed memory usage by the application is 400 MB. We will try to transfer same size file by setting transfer mode as Streamed 


<basicHttpBinding>
<binding name="basicHttpStreaming" messageEncoding="Mtom" maxReceivedMessageSize="2147483647" transferMode="Streamed">
</binding>
</basicHttpBinding>
 
Let us upload file of 88 MB size and observe the memory consumption. Below is screen shot of task manager
At any point of time client used max memory of 5 MB where as in buffered mode it was 400 MB


400 MB and 5 MB numbers may vary depend on so many other factors, but my intention here is to show you the memory consumption difference between buffered and streamed transfer mode

Download this code and toggle transfer mode between buffered and streamed and observe memory consumption

Let’s go into some more details on streaming
  1. MTOM helps with transfer overhead and processing performance, but still entire message is still loaded into memory. i.e when we upload a file of size 88MB client will load entire message into memory. Once file is loaded completely service start taking it.
  2. In Streaming as soon as message starts loading into memory, service starts receiving
  3. Streaming will lose fewer features like reliable messaging and message security
  4. Streaming is supported by NetNamedPipeBinding, NetTcpBinding and Basic HttpBinding
  5. Streaming  will expose message body as stream, but header will still buffered
  6. Following are the different options in Transfer Mode
    • Buffered – This is the default setting
    • Streamed- Streams both request and response
    • StreamedRequest – Request is streamed and response is buffered
    • StreamedResponse– Request is buffered and response is streamed
  7. Points to note on when to close stream
    • WCF will close the stream after the final read
      • For Services returning streams
      • For clients sending streams
    • We must close the stream after the final read
      • For clients reading a returned Stream
      • For services reading an incoming stream
Hope this helps
Vital.

Wednesday, May 18, 2011

Transfer large messages in WCF - Part 2

This is in continuation with Part 1. let us look into details of Message Transmission Optimization Mechanism (MTOM)

Enable MTOM for above service by setting messageEncoding = “Mtom”

<bindings>
 <basicHttpBinding>
  <binding messageEncoding="Mtom" name="largeObjects" maxReceivedMessageSize = "6500000" ></binding>
 </basicHttpBinding>
</bindings>

When we try to upload our file the bytes are converted into Base 64 encoded data in SOAP envelope.

By enabling MTOM, SOAP messages are sent as Mulitpurpose Internet Mail Extension (MIME) multipart/related content.

Here, data will not be encoded any more thus reducing processing overhead of encoding and decoding into Base 64. Another disadvantage with base 64 encoded data is that it increases the data size by approximately 33%. So by enabling MTOM we will gain this 33% size as well.

You can visualize the difference using fiddler.
Before

After


MTOM helps in improving transfer overhead and processing performance. But the entire message is still loaded into memory. Let us try to visualize this using windows task manager.

When I tried to upload a file of size 88 MB, client will load all Binary data(MTOM)/Base 64 data(Text) to memory. Once it completely loads data, the service will start receiving it and we can observe memory consumption in task manager. The client continuously increases memory consumption for our scenario till it goes up to 450 MB and then the memory consumption in service starts increasing. See below screen shot

We will see how we can improve this in the next part

Hope this helps
Vital

Friday, April 29, 2011

Transfer large messages in WCF - Part 1

Pre Requisites
                Readers must have knowledge in creating basic WCF service and must know how to host and consume a service.
Default max received message size in WCF is 65kb, so when we try to send messages more than 65kb, WCF will give errors. Let us create a basic service which will pass a message more than 65 k and see what happens and figure out what are different ways to handle large messages in WCF.

1. Create a service contract that takes byte[] as input as shown below:
[ServiceContract]
public interface IService1
{
   [OperationContract]
   void UploadFile(byte[] data);
} 
    
2.  Implement Iservice1 which will write byte[] into file on a server
public class Service1 : IService1
{
  public void UploadFile(byte[] data)
  {
    Random x = new Random();
    string fileName = string.Format(@"Test{0}.mp3", x.Next().ToString());        
    File.WriteAllBytes(fileName,data);
  }
}

3.  The app.config file for the hosting environment is given below:
  <system.serviceModel>
    <services>
      <service name="FileUpload.Service1">
        <host>
          <baseAddresses>
            <add baseAddress = "http://vital-pc:7998/FileUpload/Service1/" />
          </baseAddresses>
        </host>
        <endpoint address ="" binding="basicHttpBinding" contract="FileUpload.IService1">
        </endpoint>
        <endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange"/>
      </service>
    </services>
    <behaviors>
      <serviceBehaviors>
        <behavior>
          <serviceMetadata httpGetEnabled="True"/>
          <serviceDebug includeExceptionDetailInFaults="False" />
        </behavior>
      </serviceBehaviors>
    </behaviors>
  </system.serviceModel>

4.  Now consume this service in a client. Here we will convert a mp3 file into byte[] which is more than 65 kb and send it to service
FileStream fs = new FileStream(@"Test.mp3", FileMode.Open, FileAccess.Read);
byte[] data = new byte[fs.Length];  
fs.Read(data,0,System.Convert.ToInt32(fs.Length));  
ServiceReference1.Service1Client proxy = new ServiceReference1.Service1Client();
proxy.UploadFile(data);

5.  Make sure you enable the trace at service so that we can see the detailed error messages

6.  Make sure service is running and execute the client .

7.  As expected we got an error (look into trace at service) which says “The maximum message size quota for incoming messages (65536) has been exceeded. To increase the quota, use the MaxReceivedMessageSize property on the appropriate binding element.”. So we have to increase MaxReceivedMessageSize as below
<bindings>
 <basicHttpBinding>
  <binding name="largeObjects" maxReceivedMessageSize="6500000" ></binding>
 </basicHttpBinding>
</bindings>
    
a.  You can set maxReceivedMessageSize upto 2GB (2147483647, i.e. Int32.MaxValue)

b.  Interesting point over here is when we get above error do we need to increase maxReceivedMessageSize at service or client?
                 i. If we change maxReceivedMessageSize value at a service, it refers to the maximum message size that the service can receive. We need to change this value when we get large data from client
                 ii. If we change maxReceivedMessageSize value at client, it refers to the maximum message size that the client can receive. We need to change this value when we get large data from service
Since the data we are sending from the client exceeds the maximum permissible limit on the service, it is clear that for our problem we need to increase the maximum message size at service
             c.  maxReceivedMessageSize is a local behavior setting  i.e if you do some changes to maxReceivedMessageSize value at service and try to refresh service reference at client this value will not be updated, it will be always default value(65536) at client. This make sense with explanation in point b that explains different meanings for this property at client and service .

We have seen how we can transfer large data in WCF. But when we are working with large data, the first things that come into our mind  are memory and processing overhead. These things can be addressed using

Let us go into details of each one with examples in Part 2 and Part 3.

Hope this helps
Vital