jump to navigation

Robust Daemon Monitoring Inbox with MailKit w/ OAuth2 September 22, 2022

Posted by codinglifestyle in C#, CodeProject, Parallelism.
Tags: , , , , , , , ,
1 comment so far

It all started with an email sent to a daemon.  A windows service hosting two modules, each of which monitors an inbox for automation, dutifully ignored warnings from IT that basic authentication for O365 would be switched off in several months.  Months went by… Even though Microsoft’s announcement was 2 years ago by the time the right people were informed the deadline was just 2 months away.  Sound familiar? Unfortunately the current IMAP API doesn’t support OAuth2 authentication so would have to be replaced.  Even worse, we wasted weeks sorting out access with our Azure admin even though we had step-by-step instructions from day one.

Investigating mainstream IMAP APIs supporting OAuth2 turned up MailKit whose author was reassuringly active on Github and StackOverflow.  We quickly discovered devs everywhere were addressing this issue and there was much debate on how, or even if, it could even be done (some of this doubt from the author himself).  Thankfully after a painful couple of weeks, we were authenticating a daemon without user interaction (aka OAuth2 client credential grant flow).

When it comes to writing an API there is a spectrum in abstracting and obfuscating the inner workings from the user.  On one hand, an API written to be 1:1 with the server is less usable but may give minute control and transparency allowing for better debugging.  This path requires more ramp-up time and leaves more complexity to the user.  On the other end of the spectrum, the API does some of the heavy lifting and aims to provide a usable, easy-to-use interface.  A typical trade-off being the inner workings are a black box which might bite you in the ass down the road.

MailKit is wholeheartedly in the former camp compared with our old API.  The old one connected, there was a new email event, then disconnect when the service shuts down.  It was just overall easier to use from deleting a message to searching for new emails. For example, the email UID was part of the email object.  With MailKit this information must be queried separately because technically that’s how it’s stored on the server.  And this sets the tone for the entire experience of interacting with MailKit.

As mentioned above, even if a bit more difficult to use, it was very reassuring to see how active the author and user community are.  While porting code from the old API required a lot of rewriting there was plenty of documentation, discussion, and examples out there to answer our questions.  What was unexpected was that the server events did not work without building a full IMAP client, which reminds me of implementing a Windows message pump, to idle and process events in its own thread.  Thankfully documentation and examples, although complicated, were available to build upon.

With the preamble out of the way, we can finally talk about the code.  What follows is a C# wrapper for the MailKit API.  We can use it in two different ways.  You can simply instantiate it and execute a command with 2 lines of code.  This will automatically connect, run the IMAP command within the IMAP client thread context, and then disconnect.  Alternatively, you can use it as a long-running connection which will launch the IMAP client as a robust task which will stay connected until stopped.  This allows the use of an event the wrapper exposes to process new messages.  There is also a command queue so that code can be queued to run within the IMAP client thread context.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.CompilerServices;
using System.Threading;
using System.Threading.Tasks;
using MailKit;
using MailKit.Net.Imap;
using MailKit.Security;
using Microsoft.Identity.Client;

namespace Codinglifestyle
{
    /// <summary>
    /// IMAP client instance capable of receiving events and executing IMAP commands
    /// </summary>
    /// <seealso cref="System.IDisposable" />
    public class ImapClientEx : IDisposable
    {
        #region Member variables
        ImapClient                   _imapClient;
        IMailFolder                  _imapFolder;
        int                          _numMessages;
        CancellationTokenSource      _tokenCancel;
        CancellationTokenSource      _tokenDone;

        Queue<OnImapCommand>         _queueCommand;
        bool                         _messagesArrived;

        readonly string              _imapServer;
        readonly string              _imapUser;
        readonly string              _authAppID;
        readonly string              _authAppSecret;
        readonly string              _authTenantID;
        readonly SecureSocketOptions _sslOptions;
        readonly int                 _port;
        readonly FolderAccess        _folderAccess;
        
        protected DateTime           _dtLastConnection;

        readonly object              _lock;
        #endregion

        #region Ctor
        /// <summary>
        /// Initializes a new instance of the <see cref="ImapClientEx"/> class.
        /// </summary>
        /// <param name="userEmail">The user email account.</param>
        public ImapClientEx(string userEmail)
        {
            _queueCommand  = new Queue<OnImapCommand>();
            _numMessages   = 0;
            _lock          = new object();

            Config config  = new Config("O365 Settings");
            _authAppID     = config["App ID"];
            _authAppSecret = config.Decrypt("App Secret");
            _authTenantID  = config["Tenant ID"];
            config         = new Config("Mail Settings");
            _imapServer    = config["IMAP Server"];
            _imapUser      = userEmail;
            _sslOptions    = SecureSocketOptions.Auto;
            _port          = 993;
            _folderAccess  = FolderAccess.ReadWrite;
        }
        #endregion

        #region Public Events
        /// <summary>
        /// IMAP command delegate to be queued and executed by the IMAP thread instance.
        /// </summary>
        /// <param name="imapClient">The IMAP client.</param>
        /// <param name="imapFolder">The IMAP folder.</param>
        public delegate void OnImapCommand(ImapClient imapClient, IMailFolder imapFolder);
        /// <summary>
        /// Event indicates the IMAP client folder has received a new message.
        /// </summary>
        /// <remarks>
        /// The event is called by the IMAP thread instance.
        /// </remarks>
        public event OnImapCommand NewMessage;
        /// <summary>
        /// Fires the new message event.
        /// </summary>
        private void OnNewMessageEvent(ImapClient imapClient, IMailFolder imapFolder)
        {
            if (NewMessage != null)
                NewMessage(_imapClient, _imapFolder);
        }
        #endregion

        #region Public Methods
        /// <summary>
        /// Runs a IMAP client asynchronously.
        /// </summary>
        public async Task RunAsync()
        {
            try
            {
                //
                //Queue first-run event to load new messages since last connection (the consumer must track this)
                //
                QueueCommand(OnNewMessageEvent);
                //
                //Run command in robustness pattern asynchronously to let this thread go...
                //
                await DoCommandAsync((_imapClient, _imapFolder) =>
                {
                    //
                    //Run IMAP client async in IDLE to listen to events until Stop() is called
                    //
                    IdleAsync().Wait();
                });

                Log.Debug(Identifier + "IMAP client exiting normally.");
            }
            catch (OperationCanceledException)
            {
                //Token is cancelled so exit
                Log.Debug(Identifier + "IMAP operation cancelled...");
            }
            catch (Exception ex)
            {
                Log.Err(ex, Identifier + "RunAsync");
            }
            finally
            {
                //
                //Disconnect and close IMAP client
                //
                Dispose();
            }
        }

        /// <summary>
        /// Gets a value indicating whether this IMAP client instance is connected.
        /// </summary>
        public bool IsConnected => _imapClient?.IsConnected == true && _imapFolder?.IsOpen == true;

        /// <summary>
        /// Identifiers this instance for logging.
        /// </summary>
        public string Identifier => string.Format("IMAP {0} [{1}]: ", _imapUser, Thread.CurrentThread.ManagedThreadId);

        /// <summary>
        /// Stops this IMAP client instance.
        /// </summary>
        public void Stop()
        {
            //Cancel the tokens releasing the IMAP client thread
            _tokenDone?.Cancel();
            _tokenCancel?.Cancel();
        }

        /// <summary>
        /// Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources.
        /// </summary>
        /// <remarks>This is safe to call and then carry on using this instance as all the resources will be automatically recreated by error handling</remarks>
        public void Dispose()
        {
            //Cancel tokens
            Stop();
            //Release connection
            DisconnectAsync().Wait();
            //Release resources
            if (_imapFolder != null)
            {
                _imapFolder.MessageExpunged -= OnMessageExpunged;
                _imapFolder.CountChanged    -= OnCountChanged;
            }
            _imapFolder  = null;
            _imapClient?.Dispose();
            _imapClient  = null;
            _tokenCancel?.Dispose();
            _tokenCancel = null;
            _tokenDone?.Dispose();
            _tokenDone   = null;
        }
        #endregion

        #region IMAP Connect / Idle
        /// <summary>
        /// Connects IMAP client, authenticated with OAUTH2, and opens the Inbox folder asynchronously.
        /// </summary>
        private async Task ConnectAsync()
        {
            //Dispose of existing instance, if any.
            if (_imapClient != null)
                Dispose();
            //
            //Create IMAP client
            //
            _imapClient  = new ImapClient();
            //
            //Create a new cancellation token
            //
            _tokenCancel = new CancellationTokenSource();
            //
            //Connect to the server
            //
            Log.Debug(Identifier + "Connecting to IMAP server: " + _imapServer);
            if (!_imapClient.IsConnected)
                await _imapClient.ConnectAsync(_imapServer, _port, _sslOptions, _tokenCancel.Token);
            //
            //Authenticate
            //
            if (!_imapClient.IsAuthenticated)
            {
                //
                //Create the client application
                //
                var app       = ConfidentialClientApplicationBuilder
                                    .Create(_authAppID)
                                    .WithClientSecret(_authAppSecret)
                                    .WithAuthority(new System.Uri($"https://login.microsoftonline.com/{_authTenantID}"))
                                    .Build();
                //
                //Get the OAUTH2 token
                //
                var scopes    = new string[] { "https://outlook.office365.com/.default" };
                var authToken = await app.AcquireTokenForClient(scopes).ExecuteAsync();
                Log.Debug(Identifier + "Creating OAUTH2 tokent for {0}: {1}", _imapUser, authToken.AccessToken);
                var oauth2    = new SaslMechanismOAuth2(_imapUser, authToken.AccessToken);
                //
                //Authenticate
                //
                Log.Debug(Identifier + "Authenticating user: " + _imapUser);
                await _imapClient.AuthenticateAsync(oauth2, _tokenCancel.Token);
            }
            //
            //Open inbox
            //
            if (!_imapClient.Inbox.IsOpen)
                await _imapClient.Inbox.OpenAsync(_folderAccess, _tokenCancel.Token);
            // Note: We capture client.Inbox here because cancelling IdleAsync() *may* require
            // disconnecting the IMAP client connection, and, if it does, the `client.Inbox`
            // property will no longer be accessible which means we won't be able to disconnect
            // our event handlers.
            _imapFolder                  = _imapClient.Inbox;
            //
            //Track changes to the number of messages in the folder (this is how we'll tell if new messages have arrived).
            _imapFolder.CountChanged    += OnCountChanged;
            //Track of messages being expunged to track messages removed to work in combination with the above event.
            _imapFolder.MessageExpunged += OnMessageExpunged;
            //Track the message count to determine when we have new messages.
            _numMessages                 = _imapFolder.Count;
        }

        /// <summary>
        /// Closes the folder and disconnects IMAP client asynchronously.
        /// </summary>
        private async Task DisconnectAsync()
        {
            try
            {
                //Disconnect IMAP client
                if (_imapClient?.IsConnected == true)
                    await _imapClient.DisconnectAsync(true);
                Log.Debug(Identifier + "Disconnected.");
            }
            catch (Exception)
            {
            }
        }

        /// <summary>
        /// Idles waiting for events or commands to execute asynchronously.
        /// </summary>
        private async Task IdleAsync()
        {
            do
            {
                try
                {
                    //
                    //Run all queued IMAP commands
                    //
                    await DoCommandsAsync();
                    //
                    //Idle and listen for messages
                    //
                    await WaitForNewMessages();
                    //
                    if (_messagesArrived)
                    {
                        Log.Debug(Identifier + "New message arrived.  Queueing new message event...");
                        //
                        QueueCommand(OnNewMessageEvent);
                        //
                        _messagesArrived = false;
                    }
                }
                catch (OperationCanceledException)
                {
                    //Token is cancelled so exit
                    Log.Debug(Identifier + "IMAP Idle stopping...");
                    break;
                }
            } while (_tokenCancel != null && !_tokenCancel.IsCancellationRequested);
        }

        /// <summary>
        /// Waits for server events or cancellation tokens asynchronously.
        /// </summary>
        private async Task WaitForNewMessages()
        {
            try
            {
                Log.Debug(Identifier + "IMAP idle for 1 minute.  Connection age: {0}", DateTime.Now - _dtLastConnection);
                if (_imapClient.Capabilities.HasFlag(ImapCapabilities.Idle))
                {
                    //Done token will self-desrtruct in specified time (1 min)
                    _tokenDone = new CancellationTokenSource(new TimeSpan(0, 1, 0));
                    //
                    //Idle waiting for new events...
                    //Note: My observation was that the events fired but only after the 1 min token expired
                    //
                    await _imapClient.IdleAsync(_tokenDone.Token, _tokenCancel.Token);
                }
                else
                {
                    //Wait for 1 min
                    await Task.Delay(new TimeSpan(0, 1, 0), _tokenCancel.Token);
                    //Ping the IMAP server to keep the connection alive
                    await _imapClient.NoOpAsync(_tokenCancel.Token);
                }
            }
            catch (OperationCanceledException)
            {
                Log.Debug(Identifier + "WaitForNewMessages Idle cancelled...");
                throw;
            }
            catch (Exception ex)
            {
                Log.Warn(ex, Identifier + "WaitForNewMessages errored out...");
                throw;
            }
            finally
            {
                _tokenDone?.Dispose();
                _tokenDone = null;
            }
        }
        #endregion

        #region Command Queue
        /// <summary>
        /// Connects and performs IMAP command asynchronously.
        /// </summary>
        /// <param name="command">The IMAP comannd to execute.</param>
        /// <param name="retries">The number of times to retry executing the command.</param>
        /// <returns>Return true if the command succesfully updated</returns>
        /// <exception cref="MailKit.ServiceNotConnectedException">Will enter robustness pattern if not connected and retry later</exception>
        public async Task<bool> DoCommandAsync(OnImapCommand command, int retries = -1)
        {
            int attempts                  = 1;
            int errors                    = 0;
            int connections               = 0;
            _dtLastConnection             = DateTime.Now;
            DateTime errorStart           = DateTime.Now;
            bool bReturn                  = false;

            //Enter robustness pattern do/while loop...
            do
            {
                try
                {
                    //
                    //Connect, if not already connected
                    //
                    if (!IsConnected)
                    {
                        Log.Debug(Identifier + "Connection attempt #{0}; retries: {1}; errors: {2}; conns: {3}; total age: {4})",
                                  attempts++,
                                  (retries-- < 0) ? "infinite" : retries.ToString(),
                                  errors,
                                  connections,
                                  DateTime.Now - _dtLastConnection);
                        //
                        //Connect to IMAP
                        //
                        await ConnectAsync();
                        //Test IMAP connection
                        if (!IsConnected)
                            throw new ServiceNotConnectedException();
                        Log.Debug($"{Identifier}Server Connection: {IsConnected}");
                        //Reset connection stats
                        attempts          = 1;
                        errors            = 0; 
                        _dtLastConnection = DateTime.Now;
                        connections++;
                    }
                    //
                    //Perform command
                    //
                    Log.Debug("{0}Run IMAP command: {1}", Identifier, command.Method);
                    await Task.Run(() => command(_imapClient, _imapFolder), _tokenCancel.Token);
                    //
                    //Success: break the do/while loop and exit
                    //
                    Log.Debug(Identifier + "Command completed successfully.");
                    bReturn = true;
                    break;
                }
                catch (OperationCanceledException)
                {
                    //Token is cancelled so break the do/while loop and exit
                    Log.Debug(Identifier + "Command operation cancelled...");
                    break;
                }
                catch (Exception ex)
                {
                    //If no reries left log the error
                    if (retries == 0 && IsConnected)
                        Log.Err(ex, "{0}Error IMAP command: {1}", Identifier, command.Method);
                    //If first error since connected...
                    if (errors++ == 0)
                    {
                        //Track time since first error
                        errorStart = DateTime.Now;
                        //Reset the IMAP connection
                        Log.Debug(Identifier + "Error detected - attempt immediate reconnection.");
                        await DisconnectAsync();
                    }
                    else
                    {
                        TimeSpan errorAge = (DateTime.Now - errorStart);
                        Log.Debug(Identifier + "Connect failure (attempting connection for {0})", errorAge);

                        //Wait and try to reconnect
                        if (errorAge.TotalMinutes < 10)
                        {
                            Log.Debug(Identifier + "Cannot connect.  Retry in 1 minute.");
                            await Task.Delay(new TimeSpan(0, 1, 0), _tokenCancel.Token);
                        }
                        else if (errorAge.TotalMinutes < 60)
                        {
                            Log.Info(Identifier + "Cannot connect.  Retry in 10 minutes.");
                            await Task.Delay(new TimeSpan(0, 10, 0), _tokenCancel.Token);
                        }
                        else
                        {
                            Log.Err(ex, Identifier + "Cannot connect.  Retry in 1 hour (total errors: {0}).", errors);
                            await Task.Delay(new TimeSpan(1, 0, 0), _tokenCancel.Token);
                        }
                    }
                }
            } while (retries != 0 && _tokenCancel != null && !_tokenCancel.IsCancellationRequested);
            //
            //Return true if the command succesfully updated
            //
            return bReturn;
        }

        /// <summary>
        /// Execute the IMAP commands in the queue asynchronously.
        /// </summary>
        /// <param name="retries">The number of times to retry executing the command.</param>
        /// <returns>True if all commands in the queue are executed successfully.</returns>
        /// <remarks>Command retries do not apply to the queue which will run idefinitely until empty or cancelled</remarks>
        public async Task<bool> DoCommandsAsync(int retries = -1)
        {
            while (_queueCommand.Count > 0 && _tokenCancel != null && !_tokenCancel.IsCancellationRequested)
            {
                try
                {
                    //Peek in the command queue for the next command
                    var command = _queueCommand.Peek();
                    //
                    //Execute the Imap command
                    //
                    if (await DoCommandAsync(command, retries))
                    {
                        //If successful, dequeue and discard the command
                        lock (_lock)
                            _queueCommand.Dequeue();
                    }
                    //Reset if the command affects folder state
                    if (_imapClient.IsConnected && !_imapFolder.IsOpen)
                        _imapFolder.Open(_folderAccess);
                }
                catch (Exception ex)
                {
                    //We may be disconnected, throw to try again
                    Log.Warn(ex, Identifier + "DoCommands errored out...");
                    throw;
                }
            }

            return _queueCommand.Count == 0;
        }

        /// <summary>
        /// Queues a command to be executed by the IMAP client instance.
        /// </summary>
        /// <param name="command">The command to execute in the IMAP thread.</param>
        public void QueueCommand(OnImapCommand command)
        {
            lock (_lock)
                _queueCommand.Enqueue(command);
            //If idling, wake up and process the command queue
            _tokenDone?.Cancel();
        }
        #endregion

        #region IMAP Events
        /// <summary>
        /// Called when folder message count changes.
        /// </summary>
        /// <param name="sender">The sender.</param>
        /// <param name="e">The <see cref="EventArgs"/> instance containing the event data.</param>
        /// <remarks>CountChanged event will fire when new messages arrive in the folder and/or when messages are expunged.</remarks>
        private void OnCountChanged(object sender, EventArgs e)
        {
            var folder = (ImapFolder)sender;
            Log.Debug(Identifier + "{0} message count has changed from {1} to {2}.", folder, _numMessages, folder.Count);

            //If the folder count is more than our tracked number of messages flag and cancel IDLE
            if (folder.Count > _numMessages)
            {
                Log.Debug(Identifier + "{0} new messages have arrived.", folder.Count - _numMessages);
                // Note: This event is called by the ImapFolder (the ImapFolder is not re-entrant).
                //       IMAP commands cannot be performed here so instead flag new messages and
                //       cancel the `done` token to handle new messages in IdleAsync.
                _messagesArrived = true;
                _tokenDone?.Cancel();
            }
            //
            //Track the message count to determine when we have new messages.
            //
            _numMessages = folder.Count;
        }

        /// <summary>
        /// Called when a message is expunged (deleted or moved).
        /// </summary>
        /// <param name="sender">The sender.</param>
        /// <param name="e">The <see cref="MessageEventArgs"/> instance containing the event data.</param>
        private void OnMessageExpunged(object sender, MessageEventArgs e)
        {
            var folder = (ImapFolder)sender;
            Log.Debug(Identifier + "{0} message #{1} has been expunged.", folder, e.Index);
            //
            //Track the message count to determine when we have new messages.
            //
            _numMessages = folder.Count;
        }
        #endregion
    }
}

It is worth studying the “robustness pattern” in the DoCommandAsync code. The most likely reason for an exception to be thrown, provided your own code is well written with error handling, is due to a problem with the server connection. This pattern is meant to allow a daemon to reestablish a connection even if it takes hours to do so. The idea is that on the first error it will immediately reconnect and try again. If there is still a connection issue it will wait 1 minute between retries, then 10 minutes, and then ultimately wait an hour before trying to reconnect and run the command. There is also a way of retrying indefinitely or for a specified number of retries.

It should also be noted that, as in the author’s example, there are two cancellation tokens being used. These can be accessed via the wrapper by calling Stop or Disposing the wrapper instance. When a command is queued we’ll wake up if idling. When a server event is received we should do likewise.

First, let’s demonstrate the simple case of connecting and running an IMAP command (such as deleting an email, searching for or fetching details, or moving a message, etc). 

//Connect, perform command, and disconnect synchronously
using (var imapClient = new ImapClientEx(_imapUser))
{
    //The IMAP client will run the command async so we must Wait to ensure the connection does not close before the command is run
    imapClient.DoCommandAsync(MoveEmail, 5).Wait();
}

Notice the using statement for scoping the ImapClientEx wrapper.  This code is being executed by its own thread, when the command is run this is done in the IMAP client thread, and the pointer to the function is shunted over from one thread to the other.  The wrapper will automatically connect before running the command. While async is supported, in this case, we will wait otherwise we will dispose of our IMAP connection too soon.

private void MoveEmail(ImapClient imapClient, IMailFolder imapFolder)
{
    //Perform an action with the connected imapClient or the opened imapFolder
}

The command queue takes a delegate with parameters for the MailKit client and folder arguments. It is run by the IMAP client wrapper thread but it is the instance of your object so you have full access to member variables, etc. Again, this is a simple use case but shows how easily the client can connect and run code.

Now let’s move on to the use case where you want to have a long-standing connection to an inbox to monitor new messages.  This requires asynchronously launching and storing the IMAP client wrapper.  As the client is running it will remain connected and monitor two events as per the author’s example: inbox.CountChanged and inbox.MessageExpunged.  By handling these the wrapper can expose its single event: NewMessage.  With the IMAP client running, all we have to do is keep the instance in a member variable to queue additional IMAP commands, receive the NewMessage event, or stop the client when we are done.

        protected void ImapConnect()
        {
            // Dispose of existing instance, if any.
            if (_imapClient != null)
            {
                _imapClient.NewMessage -= IMAPProcessMessages;
                _imapClient.Stop();
                _imapClient = null;
            }

            _imapClient             = new ImapClientEx(_imapUser);
            _imapClient.NewMessage += IMAPProcessMessages;
            var idleTask            = _imapClient.RunAsync();
            _dtLastConnection       = DateTime.Now;
        }

Now it should be noted that a NewMessage event will fire once the IMAP client connects at startup.  This is because our daemon needs to be capable of shutting down and therefore must track the last processed message.  The best way to do this is to track the last UID processed.  This way, whenever the event is fired, you will just search for new UIDs since the last tracked UID was seen. 

        private void IMAPProcessMessages(ImapClient imapClient, IMailFolder imapFolder)
        {
            LogSvc.Debug(this, "IMAP: Checking emails...");
            _dtLastConnection = DateTime.Now;
            //
            //Retrieve last index from DB
            //
            if (_currentUid  == 0)
                _currentUid  = (uint)TaskEmailData.FetchLastUID(_taskType);
            LogSvc.Debug(this, "IMAP: Last email index from DB: " + _currentUid.ToString());
            //
            //Process messages since last processed UID
            //
            int currentIndex = imapFolder.Count - 1;
            if (currentIndex >= 0)
            {
                //
                //Create range from the current UID to the max
                //
                var range    = new UniqueIdRange(new UniqueId((uint)_currentUid + 1), UniqueId.MaxValue);
                //
                //Get the UIDs newer than the current UID
                //
                var uids     = imapFolder.Search(range, SearchQuery.All);
                //
                if (uids.Count > 0)
                {
                    LogSvc.Info(this, "IMAP: Processing {0} missed emails.", uids.Count);
                    foreach (var uid in uids)
                    {
                        //
                        //Get the email
                        //
                        var email = imapFolder.GetMessage(uid);
                        //
                        //Process and enqueue new message
                        //
                        ImapProcessMessage(imapClient, imapFolder, uid, email);
                    }
                    //
                    //Pulse the lock to process new tasks...
                    //
                    Pulse();
                }
                else
                {
                    LogSvc.Debug(this, "IMAP: No missed emails.");
                }
            }
            else
            {
                LogSvc.Debug(this, "IMAP: No missed emails.");
            }
        }

I won’t show you but, suffice it to say, I have one extra level of redundancy in my daemon where it tracks the connection age and simply recycles the connection after a specified amount of time of inactivity. This was done because, while it was more usable, our old IMAP API became disconnected quite regularly although it falsely reported it was still connected.

Lastly, when the daemon is being shut down for any reason, we need to Stop or Dispose to disconnect and clean up the IMAP connection. The Stop will trigger the cancellation tokens such that the IMAP task thread shuts down in its own time. Calling Dispose directly will synchronously do the same. Also, Dispose can be called repeatedly on the wrapper instance and still be safe to use as it will reconnect as necessary.

_imapClient?.Dispose();

This took a couple of weeks to write and test. My boss was cool about it being shared so I hope to save someone else the pain of writing this from scratch.  While MailKit may be on the basic end of the spectrum we built a solution that is very robust and will no doubt have a better up-time metric than before. Many thanks to the author and the MailKit user community for all the insight and knowledge necessary to have written this.

Thread Synchronization Between Worker Threads April 15, 2021

Posted by codinglifestyle in Architecture, C#, CodeProject, Parallelism.
Tags: , , , ,
add a comment

Many moons ago I wrote the last Windows service I’d ever need. It has a pluggable architecture and today there are over 20 modules doing all manner of business tasks such as sending emails, generating reports, performing automation, etc. Inside each plug-in module is a thread daemon which listens to events to orchestrate the operations of the other supporting module threads. This primarily consists of a thread which provides incoming data (what?), a thread which is the scheduler (when?), and a thread pool of worker threads which process said data (how?).

Recently a new plug-in was created to enforce global trade compliance. A bug crept up in the new worker thread initialization which surprised me because it caused the incoming data queue thread to die even though that code hadn’t changed in years. The reason was that the incoming queue fired the event and therefore it was the thread which executed the delegate (aka function pointer). This function is defined in the thread daemon class which takes care of initializing and starting a worker thread. From an error handling point of view this struck me as less than ideal. So the question was asked:

How do I get the thread daemon to be the thread which executes it’s own management functions which are triggered by events?

This got me thinking about WinForm apps whereby a worker thread must marshal execution of a delegate to the main UI thread to update the UI like changing the status bar. The pattern is to check if an invoke is required (am I the UI thread?) and if not then you use the built in SynchronizationObject to invoke the delegate in the UI thread.

        public delegate void UpdateStatusEvent(string message, params object[] args);

        public void UpdateStatus(string message, params object[] args)
        {
            if (_Status.InvokeRequired)
            {
                UpdateStatusEvent amc = new UpdateStatusEvent(UpdateStatus);
                this.Invoke(amc, new object[] { message, args });
            }
            else
            {
                //Now we are the UI thread we can actually update the status bar
                _LabelStatus.Text = string.Format(message, args);
                _Status.Refresh();
            }
        }

With this pattern in my head I thought I’d be able to implement this quite easily between my background worker threads. How wrong I was and hence this article! WinForms and WPF provide a synchronization context but there isn’t something in the framework you get for free in my case. From what I understand they both implement a classic windows message pump which is used to marshal the execution from the worker thread to the UI thread. This will become important shortly, but suffice it to say after a lot of searching a pre-baked, easy solution wasn’t on offer.

As I continued down this path I searched for ISynchronizeInvoke and IAsyncResult which primarily returned questions and examples of the above pattern. I tried in vein to find an equivalent implementation except for worker threads. Surely it wasn’t that complicated to get an event initiated in Thread A to execute in Thread B? I tried using the SynchronizationContext class but quickly discovered it’s just a base class and didn’t do the work of marshalling the execution from Thread A to Thread B. So while I went through the motions the code still executed in the wrong thread context (so why isn’t this class abstract if you must extend it to get it to work?). BTW, the way I was testing to see which thread was running was to refer to Thread.CurrentThread.ManagedThreadId.

So now I had wasted a lot of time trying to find an easy answer and had to accept some work on my part would be necessary in this scenario. What struck me was the notion of the classic windows pump being crucial for the synchronization context. Basically that it has a message queue running so a thread can enqueue a message and that message then be executed by another thread when it is able. So in the thread daemon I defined a queue and in the OnDoWork created my rudimentary “message pump”.

        private Queue<WaitCallback> _qWorkItems;
        private object _oLock;
      
        protected override void OnDoWork(DoWorkEventArgs e)
        {
            LogSvc.Debug(this, "Running");

            while (!CancellationPending)
            {
                //Critical section
                lock (_oLock)
                {
                    //This is the message pump allowing for the thread synchronization context
                    while (_qWorkItems.Count > 0)
                    {
                        LogSvc.Debug(this, "Dequeue and invoke work item on thread {0}", Thread.CurrentThread.ManagedThreadId);

                        //Dequeue next work item
                        var workItem = _qWorkItems.Dequeue();
                        
                        //Execute work item in thread daemon context
                        workItem.Invoke(null);
                    }

                    //Wait for new work items to process
                    Monitor.Wait(_oLock);
                }
            }
        }

        /// <summary>
        /// Queues a method for execution.  The method executes when the thread daemon is available.
        /// </summary>
        /// <param name="callback">The callback.</param>
        public void QueueWorkItem(WaitCallback callback)
        {
            LogSvc.Debug(this, "Enqueuing work item from event caller thread {0}", Thread.CurrentThread.ManagedThreadId);

            //Critical section
            lock (_oLock)
            {
                _qWorkItems.Enqueue(callback);
                Monitor.Pulse(_oLock);
            }
        }

I started with a Queue<Action> which allowed me to accomplish asynchronous execution. However I wanted to be able to support synchronous execution as well to support getting the return value from the delegate. So I looked at what ThreadPool.EnqueueUserWorkItem used and settled on WaitCallback.

Now we have our thread daemon setup to queue and execute operations in it’s thread context. What we need next is a synchronization context to allow the worker threads to marshal the delegate and data from their thread to the thread daemon thread. We’ll implement both ISynchronizeInvoke and IAsyncResult classes to nicely encapsulate this functionality. This will offer a test to see if an invoke is required and support both asynchronous and synchronous execution of the event delegate.

    /// <summary>
    /// Internally used by ThreadSynchronization to represent asynchronous operations
    /// </summary>
    /// <seealso cref="System.IAsyncResult" />
    class ThreadAsyncResult : IAsyncResult
    {
        /// <summary>
        /// Gets a value that indicates whether the asynchronous operation has completed.
        /// </summary>
        public bool IsCompleted { get; set; }

        /// <summary>
        /// Gets a <see cref="T:System.Threading.WaitHandle" /> that is used to wait for an asynchronous operation to complete.
        /// </summary>
        public WaitHandle AsyncWaitHandle { get; internal set; }

        object _state;
        /// <summary>
        /// Gets a user-defined object that qualifies or contains information about an asynchronous operation.
        /// </summary>
        public object AsyncState
        {
            get
            {
                if (Exception != null)
                {
                    throw Exception;
                }
                return _state;
            }
            internal set
            {
                _state = value;
            }
        }

        /// <summary>
        /// Gets a value that indicates whether the asynchronous operation completed synchronously.
        /// </summary>
        public bool CompletedSynchronously { get { return IsCompleted; } }

        /// <summary>
        /// Gets or sets the exception.
        /// </summary>
        /// <value>
        /// The exception.
        /// </value>
        internal Exception Exception { get; set; }
    }

    /// <summary>
    /// Thread synchronization context to marshal delegate and data to ThreadManager thread
    /// </summary>
    /// <seealso cref="System.ComponentModel.ISynchronizeInvoke" />
    class ThreadSynchronization : ISynchronizeInvoke
    {
        public readonly int _nExecutingContextID = 0;
        private ThreadManager _manager;
        
        /// <summary>
        /// Initializes a new instance of the <see cref="ThreadSynchronization"/> class.
        /// </summary>
        /// <param name="manager">The thread manager object</param>
        public ThreadSynchronization(ThreadManager manager)
        {
            _nExecutingContextID = Thread.CurrentThread.ManagedThreadId;
            _manager             = manager;

            Log.Debug("Synchronization context created for thread {0}", _nExecutingContextID);
        }

        /// <summary>
        /// Gets a value indicating whether the caller must call <see cref="M:System.ComponentModel.ISynchronizeInvoke.Invoke(System.Delegate,System.Object[])" /> when calling an object that implements this interface.
        /// </summary>
        public bool InvokeRequired => Thread.CurrentThread.ManagedThreadId != _nExecutingContextID;

        /// <summary>
        /// Asynchronously executes the delegate on the thread that created this object.
        /// </summary>
        /// <param name="method">A <see cref="T:System.Delegate" /> to a method that takes parameters of the same number and type that are contained in <paramref name="args" />.</param>
        /// <param name="args">An array of type <see cref="T:System.Object" /> to pass as arguments to the given method. This can be <see langword="null" /> if no arguments are needed.</param>
        /// <returns>
        /// An <see cref="T:System.IAsyncResult" /> interface that represents the asynchronous operation started by calling this method.
        /// </returns>
        public IAsyncResult BeginInvoke(Delegate method, object[] args)
        {
            var result                 = new ThreadAsyncResult();
            var manualResetEvent       = new ManualResetEvent(false);
            result.AsyncWaitHandle     = manualResetEvent;

            _manager.QueueWorkItem(delegate
            {
                try
                {
                    //Invoke the delegate and capture the return value
                    result.AsyncState  = method.DynamicInvoke(args);
                }
                catch (Exception ex)
                {
                    Log.Err(ex);
                    //Capture the exception
                    result.Exception   = ex;
                }
                finally
                {
                    //Mark complete
                    result.IsCompleted = true;
                    //Set event for anyone waiting
                    manualResetEvent.Set();
                }
            });

            return result;
        }

        /// <summary>
        /// Waits until the process started by calling <see cref="M:System.ComponentModel.ISynchronizeInvoke.BeginInvoke(System.Delegate,System.Object[])" /> completes, and then returns the value generated by the process.
        /// </summary>
        /// <param name="result">An <see cref="T:System.IAsyncResult" /> interface that represents the asynchronous operation started by calling <see cref="M:System.ComponentModel.ISynchronizeInvoke.BeginInvoke(System.Delegate,System.Object[])" />.</param>
        /// <returns>
        /// An <see cref="T:System.Object" /> that represents the return value generated by the asynchronous operation.
        /// </returns>
        public object EndInvoke(IAsyncResult result)
        {
            //If not complete then wait until done
            if (!result.IsCompleted)
            {
                result.AsyncWaitHandle.WaitOne();
            }

            //The return value of the delegate
            return result.AsyncState;
        }

        /// <summary>
        /// Synchronously executes the delegate on the thread that created this object and marshals the call to the creating thread.
        /// </summary>
        /// <param name="method">A <see cref="T:System.Delegate" /> that contains a method to call, in the context of the thread for the control.</param>
        /// <param name="args">An array of type <see cref="T:System.Object" /> that represents the arguments to pass to the given method. This can be <see langword="null" /> if no arguments are needed.</param>
        /// <returns>
        /// An <see cref="T:System.Object" /> that represents the return value from the delegate being invoked, or <see langword="null" /> if the delegate has no return value.
        /// </returns>
        public object Invoke(Delegate method, object[] args)
        {
            //Get IAsyncResult operation
            var result = BeginInvoke(method, args);
            //Wait for asynchronous operation to complete
            EndInvoke(result);
            //The return value of the delegate
            return result.AsyncState;
        }
    }

So notice that ThreadSynchronization is tied to our thread daemon object which implements QueueWortItem. You could expose access to QueueWorkItem a different way if you wish. So, at long last, we have everything setup so we’re ready to alter the events themselves. These events, located within the thread daemon class, would have executed in another worker thread’s execution context. By instantiating the ThreadSynchronization object we can test if an invoke is required and enqueue the work to execute on the thread daemon thread and even get the return result of the event.

        bool Incoming_Dequeued(object oData)
        {
            bool bReturn = false;

            //If the event is called by a thread other than thread daemon...
            if (Sync.InvokeRequired)
            {
                //Marshal delegate call and data to thread daemon context
                var result = Sync.Invoke(new Func<object, bool>(Incoming_Dequeued), new object[] { oData });
                bReturn    = TypeParser.ParseBool(result, false);

                return bReturn;
            }

            //Execute this code in the context of the thread daemon

            return bReturn;
        }

        void ThreadPool_ThreadComplete(IModuleThread sender, object oResult)
        {
            //If the event is called by a thread other than thread daemon...
            if (Sync.InvokeRequired)
            {
                //Marshal delegate call and data to thread daemon context
                Sync.Invoke(new Action<IModuleThread, object>(ThreadPool_ThreadComplete), new object[] { sender, oResult });
                return;
            }

            //Execute this code in the context of the thread daemon
        }

At last here is the pattern I was looking for all too familiar to anyone who has worked on WinForms or WPF. Now we can easily see if we’re the correct thread and if not do an asynchronous or synchronous invoke of the delegate and data. When invoking it’s easy to use an Action<TIn1, TIn2> or Func<TIn1, Tin2, TOut>, as required, to generate your delegate.

In conclusion, you can see why Microsoft didn’t have a prepackaged solution to this as they couldn’t presume the executing thread would implement a message pump in a strict fashion. They did provide ISynchronizeInvoke which also needs IAsyncResult. Just creating these objects and implementing their interfaces lays bare what you need to do. While I love how it encapsulates this functionality in a familiar manner, it’s not strictly necessary. Really just the implementation of the message pump in our executing thread along with a composite object containing the delegate, data, and a lock would be enough to marshal the required pieces across and signal the thread who fired the event when execution is complete. However, if like me, you are hunting for a best practice implementation I’m very happy with how neatly the above solution turned out in the end.

Data Optimization Using Data Request Objects Implementing IEquatable December 23, 2013

Posted by codinglifestyle in Architecture, ASP.NET, C#, CodeProject.
Tags: , , , , ,
add a comment

Anyone who has written enterprise software often knows an ideal design and a linear code path are not too common. Even if you are the architect of your application that doesn’t save you from the hoops you must jump through to connect and interact with other systems in your enterprise’s ecosphere. Such as it is in my system to do a seemingly simple task, loading addresses. While there is only one address control and one presenter, when we get to the data layer there are many different code paths depending on the type of address, the selected company, sales areas, and backend systems. For a large order with hundreds of quotes there is theoretically several hundred address controls to be populated efficiently throughout the ordering process.

Before we get going on optimization let’s start with some basics. We have data consumers who want data. These consumers may represent my address control, a table, or any number of components that need data. In addition, there may be hundreds or thousands of them. We can’t allow each instance to simply call our data layer individually or we’ll cripple the system. These data requests need to be managed and optimized.

We are going to encapsulate our data request in an object. If you’re data layer function signature takes 3 parameters simply move these to your new data request object and later we’ll rewrite your data function to take a List<DataRequest> instead of 3 parameters. Of course you may have many parameters or complex objects you need to pass, all the better to encapsulate them! So now we have an object which contains all the information we need to ultimately call the data layer.

When you have hundreds or thousands of data requests there is a very good chance that many of those requests are for the same data. That’s what we’re after here is minimizing the number of calls for actual data. Of course, due to how legacy data functions may be written they may be too narrow in scope. Some queries, for example, may be filtered based on a function parameter which then might require multiple calls to get the complete data required across all data requests. This is the kind of analysis you will need to perform on your own to perhaps bring back the larger data set, cache it, and return pieces of it to individual data requests. One of the great advantages of encapsulating your data requests is analysing them and being able to better satisfy them by rewriting your data layer functions.

Next we must design our controls and other data consumers to be patient. Instead of making a call to get some data which is immediately fulfilled they will instead register a data request. This will give the hundreds or thousands of other data consumers a chance to register their data requests.

Once the registration window is closed we can now trigger our service or presenter to make the necessary data layer call. As eluded to above, we will pass the complete list of data requests to the data layer. We will then have the opportunity to optimize the data requests to minimize the number of actual data calls made and to make them in bulk. This is the part where you might start worrying how to tackle this gargantuan task. What if I told you I could optimize your data requests in just a few lines of code?

//Create data sets of like requests  (minimized data requests)
//
Dictionary<DataLoadAddressesEntity, List> requestsEx = new Dictionary<DataLoadAddressesEntity, List>();
foreach (DataLoadAddressesEntity request in requests)
{
    if (!requestsEx.ContainsKey(request))
        requestsEx.Add(request, new List());
    else
        requestsEx[request].Add(request);
}

That wasn’t so hard, was it? Now I have a dictionary whose keys represent the minimized number of data calls truly necessary. I call these prime data requests and they are the keys in the dictionary. Each prime data request may then be used to populate the list of equal data requests which are held in the values of the dictionary. So once the prime data request is satisfied we merely need to copy the results across the values in the data set:

//////////////////////////////////////////////////////
//Copy prime request results reference across data set
//
requestsEx[requestPrime].ForEach(r => r.Results = requestPrime.Results);

You might notice that I’ve included a Results property in my data request object. The great thing about encapsulating our request in an object is how handy it is to add more properties to keep everything together. Keep in mind that we are merely copying a reference of the prime request’s results across all like data requests. Therefore, changing one affects all the others, which makes sense but must be understood to not be dangerous. Some developers can go many years without really considering what reference types are so make sure to mentor your team of the basics of value vs reference types. Coming from C++ and the wondrous pointer I take full advantage of references as you will see in my final summary below.

So you must be wondering what voodoo magic I’m using to optimize the data set so easily. Did you read the title? To know if one data request is equal to another it is up to us to implement IEquatable<DataRequest> and override GetHashCode. This is the voodoo that allows us to use Dictionary.ContainKey(datarequest) singling out a prime data request from the secondary data requests. So, how do we decide if one request is equal to another?

With so many permutations and variables in the data layer where does one start? There is no easy answer for this one. It is time for some analysis to boil down what exactly makes one data request different from another. This is the hardest part of the exercise. I started with a spreadsheet, looked at all the variables each code path required, and developed a matrix. I was able to eliminate many of the variables which were the same no matter what type of request it was (CompanyID for example). What appeared an arduous task boiled down to just a few criteria to differentiate requests from one another. Of course, it took hours of eliminating unused variables, proving assumptions that other variables were always equal, and cleaning up the code in order to see the light through the reeds.

Once your analysis is done you now know how to tell if one data request is equal to another so we don’t waste resources making the same call twice. Implementing IEquatable<DataRequest> will have you implementing Equals in your data request object where the comparing type is another data request:

public bool Equals(DataLoadAddressesEntity other)

For each criteria from your analysis, let’s assume we have a property in your data request object. For each criteria, a comparison of of this.Property != other.Property means you return false. If the other data request’s criteria are the same you are both after the same data. So if you fall through all the criteria comparisons return true and you now have one less data call to make.

You must repeat the same logic, in principle, for the GetHashCode override. Instead of comparing the search criteria, this time you are adding up the criteria’s hash codes. So much like above, if you have 2 data requests which need the same data you should also have 2 hash codes which are equal. In this way you can use the dictionary, as above, to optimize the data requests.

Although the criteria that pertains to your data requests will differ I will show mine here as I love seeing examples:

#region IEquatable Members
public bool Equals(PartnerFunctionSearchEntity other)
{
    if (!this.AddressType.Equals(other.AddressType)) return false;
    if (!this.SoldToId.Equals(other.SoldToId))       return false;
    if (!this.SalesArea.Equals(other.SalesArea))     return false;

    return SearchCriteria.DictionaryEqual(other.SearchCriteria);
}

public override int GetHashCode()
{
    unchecked  //overflow is ok, just wrap
    {
        int hash        = 17;
        const int prime = 31;  //Prime numbers

        hash = hash * prime + AddressType.ToString().GetHashCode();
        if (!string.IsNullOrEmpty(SalesArea))
            hash = hash * prime + SalesArea.GetHashCode();
        if (!string.IsNullOrEmpty(SoldToId))
            hash = hash * prime + SoldToId.GetHashCode();

        foreach (KeyValuePair<EAddressSearchCriteria, string> keyvalue in SearchCriteria)
            hash = hash * prime + keyvalue.GetHashCode();

        return hash;
    }
}
#endregion

You may be wondering where the best place to put the various parts of this solution. I would suggest a service layer which sits between the data consumers and the data layer. In my case with many instances of an address control I placed it in the control’s presenter. As there is a 1:1 relationship between control and presenter the latter contains a member variable which is the data request. On registration it contains only the criteria necessary to get the data. I am using the per request cache (HttpContext.Current.Items) to store my List<DataRequest> where all registered data requests are accumulating.

Remember, my presenter only holds a reference to it’s _Request member variable… the same reference which is in the data request queue and the same reference to which the results will be assigned.

Once registration closes the data layer call is triggered with the list of data requests. The optimization happens here, nearest the source, so as not be repeated. Once the requests are optimized and the actual data calls are made the _Request.Results still held in the presenter’s member variable will be populated and are ready to set to the view for display.

Passing Anonymous Types with Dynamic Lists October 1, 2013

Posted by codinglifestyle in C#, CodeProject, Javascript.
Tags: , ,
1 comment so far

Recently I was rewriting a function with a large method signature which took several arrays as parameters.  As you might guess, the index across these arrays was assumed to be the same which is error-prone.  I thought we should really encapsulate all this information in to a class or struct and pass in a single list instead.  Then I stopped myself as there was no use for this class beyond this one-time call to another function.

Then I thought, I know, I could use an anonymous type instead.

var datum = new { a = myData.a, b = myData.b, /* c, d, ..., */ z = myData.z };

This seems like a reasonable approach and exactly what throw-away anonymous types are for.  Then I tried to add this to a list.  Hmm, with a little creativity I was able to overcome this… except it only worked within the scope of the same function.  Well, I could have passed a list of objects, but what would I cast them to?

I really thought I’d have to cave in and just create the new class but then there was one option I hadn’t considered: dynamic.  Like JavaScript, this frees you from the restriction of static typing which was inhibiting a simple notion to pass a list of anonymous types to another function.  Now our list definition looks like this:

var data = new List<dynamic>();

Now this means I could really add in anything and its evaluation will be resolved at runtime.  So we’re finally free to use our anonymous class however we want.  We could even code access to properties which we aren’t supplying and code will compile (however you’ll get a nasty surprise at runtime).

protected void Bob()
{
    List<dynamic> data = new List<dynamic>();

    //
    //Assume lots of processing to build up all these arguments
    //
    data.Add(new { a = 1, b = "test" });

    Fred(data);
}

private void Fred(List<dynamic> data)
{
    //Fred processing logic
    foreach (var datum in data)
    {
        Trace.WriteLine(String.Format("{0}: {1}", datum.a, datum.b));
    }
}

The more evolved my client coding becomes the more restraining statically typed languages can feel.  With the dynamic keyword C# is one step ahead of me allowing a future of amazing code determined by an unlimited number of factors at runtime.

Dictionary Extensions: Define useful extensions to play safe January 18, 2012

Posted by codinglifestyle in C#, CodeProject.
Tags: , , , , , ,
add a comment
   if (searchCriteria.ContainsKey(key) &&
       !string.IsNullOrEmpty(searchCriteria[key]))
       searchTerm = searchCriteria[key];

Ever have a dictionary or similar data structure and your code has many repeated checks to pull the value when in reality you’d be happy with a default value like null or string.Empty? Well, consider the following extension to Dictionary:

    public static class DictionaryExtensions
    {
        public static TValue GetSafeValue<TKey, TValue>(this Dictionary<TKey, TValue> dictionary, TKey key)
        {
            TValue result = default(TValue);
            dictionary.TryGetValue(index, out result);
            return result;
        }
    }

Let’s you do:

    Dictionary bob = new Dictionary();
    string safe = bob.GetSafeValue(100);
    System.Diagnostics.Trace.WriteLine(safe);

where safe defaults to “” as it hasn’t been added. Stop! I know what you’re going to say and I thought of that too. You can control the default value as well:

    public static class DictionaryExtensions
    {
        /// <summary>
        /// Gets the safe value associated with the specified key.
        /// </summary>
        /// <typeparam name="TKey">The type of the key.</typeparam>
        /// <typeparam name="TValue">The type of the value.</typeparam>
        /// <param name="dictionary">The dictionary.</param>
        /// <param name="key">The key of the value to get.</param>
        public static TValue GetSafeValue<TKey, TValue>(this Dictionary<TKey, TValue> dictionary, TKey key)
        {
            return dictionary.GetSafeValue(key, default(TValue));
        }

        /// <summary>
        /// Gets the safe value associated with the specified key.
        /// </summary>
        /// <typeparam name="TKey">The type of the key.</typeparam>
        /// <typeparam name="TValue">The type of the value.</typeparam>
        /// <param name="dictionary">The dictionary.</param>
        /// <param name="key">The key of the value to get.</param>
        /// <param name="defaultValue">The default value.</param>
        public static TValue GetSafeValue<TKey, TValue>(this Dictionary<TKey, TValue> dictionary, TKey key, TValue defaultValue)
        {
            TValue result;
            if (key == null || !dictionary.TryGetValue(key, out result))
                result = defaultValue;
            return result;
        }
    }

Let’s you do:

   Dictionary bob = new Dictionary();
   string safe = bob.GetSafeValue(100, null);
   System.Diagnostics.Trace.WriteLine(safe);

where safe is null.

There’s obviously something wrong with me because I still think this stuff is cool.

I’m developing a nice little set of extensions at this point.  Often it seems like overkill to encapsulate handy functions like these in a class. I had started by deriving a class from Dictionary<TKey, TValue> but changed over to the above.

ScriptArguments: An easy way to programmatically pass arguments to script from codebehind January 13, 2012

Posted by codinglifestyle in AJAX, ASP.NET, C#, CodeProject, Javascript.
Tags: , , , , ,
1 comment so far

During my on-going adventures AJAXifying a crusty old business app I have been using a methodology by which most client events are setup in codebehind. The reason for this is I have easy access to my client ids, variables, and resources in codebehind. By constructing the script function calls at this stage, I can avoid messy and fragile in-line code. What I am endeavouring to do is remove all script from the markup itself. So instead of having MyPage.aspx with script mixed with markup I have MyPage.js and all functions here. Separate js files avoid fragile in-line code which only fails at runtime, can’t be refactored, and doesn’t play as nice with the debugger. Besides, separation of markup and script is good!

The downside to setting up all this script in the codebehind is it didn’t take long for the number of arguments to grow and become unruly. My script function signature looked like this:

function fnAddressChange(ddId, labelId, checkId, sameAsId, hidSelectId, hidSameAsId, onSelectEvent)

And in the codebehind I had this:

string selectArgs     = string.Format("'{0}', '{1}', '{2}', '{3}', '{4}', '{5}'", _DropDownAddress.ClientID, _LabelAddress.ClientID, _RowSameAs.ChildClientID, (SameAs && _SameAsAddress != null) ? _SameAsAddress.LabelControl.ClientID : "-1", _HiddenSelectedID.ClientID, _HiddenSameAs.ClientID);

string selectScript   = string.Format("fnAddressSelect({0}); ", selectArgs);
string changeScript   = string.Format("fnAddressChange({0}, '{1}'); ", selectArgs, OnClientSelect);

We can see selectArgs is getting out of control. Not only is it getting ridiculous to add more to it, the function signature in script is getting huge and the ordering is easier to mess up. So I came up with this solution:

ScriptArguments args = new ScriptArguments ();
args.Add("ddId", _DropDownAddress.ClientID);
args.Add("labelId", _LabelAddress.ClientID);
args.Add("checkId", _RowSameAs.ChildClientID);
args.Add("sameAsId", (SameAs && _SameAsAddress != null) ? _SameAsAddress.LabelControl.ClientID : "-1");
args.Add("hidSelectId", _HiddenSelectedID.ClientID);
args.Add("hidSameAsId", _HiddenSameAs.ClientID);

Not only is the codebehind cleaner but I don’t have to worry about string.Format or the order in which I add arguments in. The resulting script generated is:

args.ToString()
"{ ddId : 'ctl00__ContentMain__ControlOrderSoldTo__AddressSoldTo__DropDownAddress', labelId : 'ctl00__ContentMain__ControlOrderSoldTo__AddressSoldTo__LabelAddress', checkId : 'ctl00__ContentMain__ControlOrderSoldTo__AddressSoldTo__RowSameAs_FormField_CheckBox', sameAsId : '-1', hidSelectId : 'ctl00__ContentMain__ControlOrderSoldTo__AddressSoldTo__HiddenSelectedID', hidSameAsId : 'ctl00__ContentMain__ControlOrderSoldTo__AddressSoldTo__HiddenSameAs' }"

This is a javascript Object with a property per key set to the corresponding value. So in script I only need to take in one argument, the argument object. I can then access every piece of information inserted in to ScriptArguments via the correct key:

function fnAddressIsReadOnly(args) {
     alert(args.ddId);
     alert(args.labelId);
}

Will alert me with “ctl00__ContentMain__ControlOrderSoldTo__AddressSoldTo__DropDownAddress” and “ctl00__ContentMain__ControlOrderSoldTo__AddressSoldTo__LabelAddress”.

The great thing is how simple this was to implement:

public class ScriptArguments : Dictionary<string, string>
{
    public override string ToString()
    {
        StringBuilder script = new StringBuilder("{ ");
        this.Keys.ToList().ForEach(key => script.AppendFormat("{0} : '{1}', ", key, this[key]));
        script.Remove(script.Length - 2, 2);
        script.Append(" }");
        return script.ToString();
    }
}

This simple class solves a simple problem. I hope you find it useful.

FindControl: Recursive DFS, BFS, and Leaf to Root Search with Pruning October 24, 2011

Posted by codinglifestyle in ASP.NET, C#, CodeProject, jQuery.
Tags: , , , , , , ,
add a comment

I have nefarious reason for posting this. It’s a prerequisite for another post I want to do on control mapping within javascript when you have one control which affects another and there’s no good spaghetti-less way to hook them together. But first, I need to talk about my nifty FindControl extensions. Whether you turn this in to an extension method or just place it in your page’s base class, you may find these handy.

We’ve all used FindControl and realized it’s a pretty lazy function that only searches its direct children and not the full control hierarchy. Let’s step back and consider what we’re searching before jumping to the code. What is the control hierarchy? It is a tree data structure whose root node is Page. The most common recursive FindControl extension starts at Page or a given parent node and performs a depth-first traversal over all the child nodes.

Depth-first search
Search order: a-b-d-h-e-i-j-c-f-k-g

/// <summary>
/// Recurse through the controls collection checking for the id
/// </summary>
/// <param name="control">The control we're checking</param>
/// <param name="id">The id to find</param>
/// <returns>The control, if found, or null</returns>
public static Control FindControlEx(this Control control, string id)
{
    //Check if this is the control we're looking for
    if (control.ID == id)
        return control;

    //Recurse through the child controls
    Control c = null;
    for (int i = 0; i < control.Controls.Count && c == null; i++)
        c = FindControlEx((Control)control.Controls[i], id);

    return c;
}

You will find many examples of the above code on the net. This is the “good enough” algorithm of choice. If you have ever wondered about it’s efficiency, read on. Close you’re eyes and picture the complexity of the seemingly innocent form… how every table begets rows begets cells begets the controls within the cell and so forth. Before long you realize there can be quite a complex control heirarchy, sometimes quite deep, even in a relatively simple page.

Now imagine a page with several top-level composite controls, some of them rendering deep control heirachies (like tables). As the designer of the page you have inside knowledge about the layout and structure of the controls contained within. Therefore, you can pick the best method of searching that data structure. Looking at the diagram above and imagine the b-branch was much more complex and deep. Now say what we’re trying to find is g. With depth-first you would have to search the entiretly of the b-branch before moving on to the c-branch and ultimately finding the control in g. For this scenario, a breadth-first search would make more sense as we won’t waste time searching a complex and potentially deep branch when we know the control is close to our starting point, the root.

Breadth-first search

Search order: a-b-c-d-e-f-g-h-i-j-k

/// <summary>
/// Finds the control via a breadth first search.
/// </summary>
/// <param name="control">The control we're checking</param>
/// <param name="id">The id to find</param>
/// <returns>If found, the control.  Otherwise null</returns>
public static Control FindControlBFS(this Control control, string id)
{
    Queue<Control> queue = new Queue<Control>();
    //Enqueue the root control            
    queue.Enqueue(control);

    while (queue.Count > 0)
    {
        //Dequeue the next control to test
        Control ctrl = queue.Dequeue();
        foreach (Control child in ctrl.Controls)
        {
            //Check if this is the control we're looking for
            if (child.ID == id)
                return child;
            //Place the child control on in the queue
            queue.Enqueue(child);
        }
    }

    return null;
}

Recently I had a scenario where I needed to link 2 controls together that coexisted in the ItemTemplate of a repeater. The controls existed in separate composite controls.

In this example I need to get _TexBoxPerformAction’s ClientID to enable/disable it via _ChechBoxEnable. Depending on the size of the data the repeater is bound to there may be hundreds of instances of the repeater’s ItemTemplate. How do I guarantee I get the right one? The above top-down FindControl algorithms would return he first match of _TextBoxPerformAction, not necessarily the right one. To solve this predicament, we need a bottom-up approach to find the control closest to us. By working our way up the control hierarchy we should be able to find the textbox within the same ItemTemplate instance guaranteeing we have the right one. The problem is, as we work our way up we will be repeatedly searching an increasingly large branch we’ve already seen. We need to prune the child branch we’ve already seen so we don’t search it over and over again as we work our way up.

To start we are in node 5 and need to get to node 1 to find our control. We recursively search node 5 which yields no results.

Next we look at node 5’s parent. We’ve already searched node 5, so we will prune it. Now recursively search node 4, which includes node 3, yielding no results.

Next we look at node 4’s parent. We have already searched node 4 and its children so we prune it.

Last we recursively search node 2, which includes node 1, yielding a result!

So here we can see that pruning saved us searching an entire branch repeatedly. And the best part is we only need to keep track of one id to prune.

/// <summary>
/// Finds the control from the leaf node to root node.
/// </summary>
/// <param name="ctrlSource">The control we're checking</param>
/// <param name="id">The id to find</param>
/// <returns>If found, the control.  Otherwise null</returns>
public static Control FindControlLeafToRoot(this Control ctrlSource, string id)
{
    Control ctrlParent = ctrlSource.Parent;
    Control ctrlTarget = null;
    string pruneId = null;

    while (ctrlParent != null &&
           ctrlTarget == null)
    {
        ctrlTarget = FindControl(ctrlParent, id, pruneId);
        pruneId = ctrlParent.ClientID;
        ctrlParent = ctrlParent.Parent;
    }
    return ctrlTarget;
}

/// <summary>
/// Recurse through the controls collection checking for the id
/// </summary>
/// <param name="control">The control we're checking</param>
/// <param name="id">The id to find</param>
/// <param name="pruneClientID">The client ID to prune from the search.</param>
/// <returns>If found, the control.  Otherwise null</returns>
public static Control FindControlEx(this Control control, string id, string pruneClientID)
{
    //Check if this is the control we're looking for
    if (control.ID == id)
        return control;

    //Recurse through the child controls
    Control c = null;
    for (int i = 0; i < control.Controls.Count && c == null; i++)
    {
        if (control.Controls[i].ClientID != pruneClientID)
            c = FindControlEx((Control)control.Controls[i], id, pruneClientID);
    }

    return c;
}

Now we have an efficient algorithm for searching leaf to root without wasting cycles searching the child branch we’ve come from. All this puts me in mind jQuery’s powerful selection capabilities. I’ve never dreamed up a reason for it yet, but searching for a collection of controls would be easy to implement and following jQuery’s lead we could extend the above to search for far more than just an ID.

Custom Attributes with Extension Methods: Resource Key July 4, 2011

Posted by codinglifestyle in C#, CodeProject.
Tags: , , , , ,
add a comment

I picked up this technique in my last job to use a custom attribute to contain a resource key. The biggest benefit was all the enums in the system used this attribute which provided a way to translate that enum to text. Take a look at a sample enum:

public enum Mode
{
    [AttributeResourceKey("lblInvalid")]
    Invalid,
    [AttributeResourceKey("lblReview")]
    Review,
    [AttributeResourceKey("lblCheckout")]
    Checkout,
    [AttributeResourceKey("lblOrdered")]
    Ordered
}

Each enum uses the AttributeResourceKey to specify the resource key defined in the resx file. Combined with an extension method we can extend the enum itself to allow us to execute the following:

public void DoOperation(Mode mode)
{
    Log.Info(GetResourceString(mode.ResourceKey()));
    ...
}

The C++ head in me thinks, “why are we using reflection when a static function in a helper class could contain a switch statement to convert the enum to the resource key?”.  Technically this is sufficient and faster.  However, the C# head in me loves the idea that the enum and the resource key are intimately tied together in the same file. There is no helper function to forget to update.  The penalty of reading an attribute is a small price to pay to keep the enum and resource key together in order to increase overall maintainability.

So the first thing I am going to do is define a simple interface for my custom attributes.

public interface IAttributeValue<T>
{
    T Value { get; }
}

All this interface does is define that the custom attribute class itself will define a property called Value of type T. This will be useful when using the generic method, below, for pulling the attribute. Next we define the custom attribute class itself.

    public sealed class AttributeResourceKey : Attribute, IAttributeValue<string>
    {
        private string _resourceKey;
        public AttributeResourceKey(string resourceKey)
        {
            _resourceKey = resourceKey;
        }

        #region IAttributeValue<string> Members
        public string Value
        {
            get { return _resourceKey; }
        }
        #endregion
    }

Notice how simple the above class is. We have a constructor taking a string and a property called Value which returns said string. Now let’s look at the generic method for pulling the attribute.

    public static class AttributeHelper
    {
        /// <summary>
        /// Given an enum, pull out its attribute (if present)
        /// </summary>
        public static TReturn GetValue<TAttribute, TReturn>(object value)
        where TAttribute: IAttributeValue<TReturn>
        {
            FieldInfo fieldInfo = value.GetType().GetField(value.ToString());
            object[] attribs    = fieldInfo.GetCustomAttributes(typeof(TAttribute), false);
            TReturn returnValue = default(TReturn);

            if (attribs != null && attribs.Length > 0)
                returnValue = ((TAttribute)attribs[0]).Value;

            return returnValue;
        }
    }

The code above is the heart of code. It uses generics so you need only define this code once in a static class. By passing the attribute and return type we can extract our Value defined by IAttributeValue<TReturn>.  Using the where constraint on TAttribute allows the generic method to know this type defines a property called Value of type TReturn.  This exposes the true power of generics as without this constraint the method could only presume TAttribute is nothing more than an object.  This might tempt you to wrongly cast TAttribute in order to access it’s properties inviting an exception only seen at runtime.

Now to define our extension method, to be placed in a common namespace, to extend all enums with the ResourceKey() method.

    public static class EnumerationExtensions
    {
        /// <summary>
        /// Given an enum, pull out its resource key (if present)
        /// </summary>
        public static string ResourceKey(this Enum value)
        {
            return AttributeHelper.GetValue<AttributeResourceKey, string>(value);
        }
    }

Thanks to the generic attribute helper the above extension method looks trivial. We simply use the helper to return the resource key and now we’ve extended all our enums to have this useful property.

Understanding BackgroundWorker and Encapsulating your own Thread Class February 22, 2011

Posted by codinglifestyle in C#, CodeProject, Parallelism.
Tags: , , , , , , ,
add a comment

You may have come across this page if you were searching for any of the following:

  • BackgroundWorker events not firing
  • BackgroundWorker RunWorkerCompleted event not firing
  • BackgroundWorker threads frozen
  • Encapsulate thread class

Yesterday my web page was launching several worker threads and waiting for them to return to amalgamate the results in to a single data set to bind to a grid. Launching several worker threads and waiting for a join is a common pattern. To nicely encapsulate the thread itself I derived a class from BackgroundWorker. BackgroundWorker has many advantages such as using an event model, thread pool, and all the thread plumbing built right in. All you have to do is override OnDoWork and off you go. The RunWorkerCompleted event was used, in conjunction with a lock, to collect the data once the worker thread finished.

Everything looked good but for some reason the event never fired. The problem was that I had gotten myself in to a deadlock scenario. The expectation is that when the event fires the delegate method will run in the context of the thread which fired it. If this were true, this would have allowed my synchronization logic to operate normally and not deadlock. The reality is that BackgroundWorker goes out of its way to run this event in the calling thread’s identity. It did this so when using BackgroundWorker in conjunction with the UI no invoke will be required (an exception will be thrown if a thread tries to touch the UI’s controls requiring you to check InvokeRequired).

When in doubt, use something like this to check the identity of the thread executing the code:

Trace.WriteLine(string.Format(“Thread id {0}”, System.Threading.Thread.CurrentThread.ManagedThreadId));

Once putting the above trace in the code I would clearly see that the identity of my main thread was identical to the thread identity in the RunWorkerCompleted event. Once the code tried to aquire the lock it was all over.

So the solution in my case was not to use the RunWorkerCompleted event. I added an alternative event to my thread class and called that at the end of OnDoWork. The event executed in the context of the thread, as expected, and my synchronization logic worked fine. But I couldn’t help feeling it was a bit of a kludge and pondered whether I should be deriving from BackgroundWorker at all. However, what’s the alternative? There really aren’t other alternatives to BackgroundWorker built in to the framework but it is easy to create your own.  See below:

 /// <summary>
 /// Abstract base class which performs some work and stores it in a data property
 /// </summary>
 /// <typeparam name="T">The type of data this thread will procure</typeparam>
 public abstract class ThreadBase<T>
 {
 #region Public methods
 /// <summary>
 /// Does the work asynchronously and fires the OnComplete event
 /// </summary>
 public void DoWorkAsync()
 {
     DoWorkAsync(null);
 }

 /// <summary>
 /// Does the work asynchronously and fires the OnComplete event
 /// </summary>
 /// <param name="arguement">The arguement object</param>
 public void DoWorkAsync(object arguement)
 {
 ThreadPool.QueueUserWorkItem(DoWorkHelper, arguement);
 }

 /// <summary>
 /// Does the work and populates the Data property
 /// </summary>
 public void DoWork()
 {
 DoWork(null);
 }

 /// <summary>
 /// Does the work and populates the Data property
 /// </summary>
 /// <param name="arguement">The arguement object</param>
 /// <remarks>
 /// Can be called to run syncronously, which doesn't fire the OnComplete event
 /// </remarks>
 public abstract void DoWork(object arguement);
 #endregion

#region Private methods
 private void DoWorkHelper(object arguement)
 {
 DoWork(arguement);
 if (OnComplete != null)
     OnComplete.Invoke(this, Data);
 }
 #endregion

#region Properties
 public T Data { get; protected set; }
 #endregion

#region Events

 /// <summary>
 /// Delegate which is invoked when the thread has completed
 /// </summary>
 /// <param name="thread">The thread.</param>
 /// <param name="data">The data.</param>
 public delegate void ThreadComplete(ThreadBase<T> thread, T data);

 /// <summary>
 /// Occurs when the thread completes.
 /// </summary>
 /// <remarks>This event operates in the context of the thread</remarks>
 public event ThreadComplete OnComplete;
 #endregion
 }

My generic ThreadBase class is a lightweight baseclass substitute for BackgroundWorker providing the flexibility to call it synchronously or asynchronously, a generically typed Data property, and an OnComplete event. The OnComplete will execute in the thread’s context so synchronization of several threads won’t be a problem. Take a look at it in action:

 public class MyThread : ThreadBase<DateTime>
 {
 public override void DoWork(object arguement)
 {
 Trace.WriteLine(string.Format("MyThread thread id {0}", System.Threading.Thread.CurrentThread.ManagedThreadId));

Data = DateTime.Now;
 }
 }

What a nicely encapsulated thread! Below we can see how cleanly a MyThread can be used:

 MyThread thread = new MyThread();
 thread.OnComplete += new ThreadBase<DateTime>.ThreadComplete(thread_OnComplete);
 thread.DoWorkAsync();

 void thread_OnComplete(ThreadBase<DateTime> thread, DateTime data)
 {
 Trace.WriteLine(string.Format("Complete thread id {0}: {1}", Thread.CurrentThread.ManagedThreadId, data));
 }

Then I got to thinking what if I wanted the best of both worlds? Thanks to reflector I found out how BackgroundWorker’s RunWorkerCompleted event executes in the context of the calling thread. My generic ThreadBaseEx class offers two events: OnCompleteByThreadContext and OnCompleteByCallerContext.

 /// <summary>
 /// Abstract base class which performs some work and stores it in a data property
 /// </summary>
 /// <typeparam name="T">The type of data this thread will procure</typeparam>
 public abstract class ThreadBaseEx<T>
 {
 #region Private variables
 private AsyncOperation _asyncOperation;
 private readonly SendOrPostCallback _operationCompleted;
 #endregion

#region Ctor
 public ThreadBaseEx()
 {
 _operationCompleted = new SendOrPostCallback(AsyncOperationCompleted);
 }
 #endregion

#region Public methods
 /// <summary>
 /// Does the work asynchronously and fires the OnComplete event
 /// </summary>
 public void DoWorkAsync()
 {
 DoWorkAsync(null);
 }

 /// <summary>
 /// Does the work asynchronously and fires the OnComplete event
 /// </summary>
 /// <param name="arguement">The arguement object</param>
 public void DoWorkAsync(object arguement)
 {
 _asyncOperation = AsyncOperationManager.CreateOperation(null);
 ThreadPool.QueueUserWorkItem(DoWorkHelper, arguement);
 }

 /// <summary>
 /// Does the work and populates the Data property
 /// </summary>
 public void DoWork()
 {
 DoWork(null);
 }

 /// <summary>
 /// Does the work and populates the Data property
 /// </summary>
 /// <param name="arguement">The arguement object</param>
 /// <remarks>
 /// Can be called to run syncronously, which doesn't fire the OnComplete event
 /// </remarks>
 public abstract void DoWork(object arguement);
 #endregion

#region Private methods
 private void DoWorkHelper(object arguement)
 {
 DoWork(arguement);
 if (OnCompleteByThreadContext != null)
 OnCompleteByThreadContext.Invoke(this, Data);
 _asyncOperation.PostOperationCompleted(this._operationCompleted, arguement);
 }

private void AsyncOperationCompleted(object arg)
 {
 OnCompleteByCallerContext(this, Data);
 }
 #endregion

#region Properties
 public T Data { get; protected set; }
 #endregion

#region Events
 /// <summary>
 /// Delegate which is invoked when the thread has completed
 /// </summary>
 /// <param name="thread">The thread.</param>
 /// <param name="data">The data.</param>
 public delegate void ThreadComplete(ThreadBaseEx<T> thread, T data);

 /// <summary>
 /// Occurs when the thread completes.
 /// </summary>
 /// <remarks>This event operates in the context of the worker thread</remarks>
 public event ThreadComplete OnCompleteByThreadContext;

 /// <summary>
 /// Occurs when the thread completes.
 /// </summary>
 /// <remarks>This event operates in the context of the calling thread</remarks>
public event ThreadComplete OnCompleteByCallerContext;
 #endregion
 }

Your encapsulated thread will be the same as above, but now with two events allowing either scenario, depending on what suits.

ASP.Net Windows Authentication: Authorization by Group January 20, 2011

Posted by codinglifestyle in ASP.NET, C#, CodeProject, Security.
Tags: , , , , , , , ,
5 comments

Often in this line of work it’s the simple things that take the longest time. A seemingly simple question came up yesterday on how to lock access to a customer’s website to a specific Windows group. There are a couple of simple gotchas worth documenting in a relatively simple solution presented below.

First, let’s start with the basics. By default ASP.NET executes code using a fixed account. Assuming you are using IIS 6 or greater, the identity is specified in the application pool. However, if we set impersonation to true ASP.NET assumes the user’s identity. Combined with Windows authentication, our code will run within the context of the user’s Windows identity.

To achieve this, in the web.config we set authentication to Windows and impersonate to true. Now we will have an authenticated Windows user, we next need to focus on authorization or what rights and restrictions apply to that user. Our requirement in this case is simple, if you belong to a specified Windows group you have access, otherwise you do not. When using Windows authentication, roles within ASP.NET translate to Windows groups. To allow a specific Windows group, allow that role within the authorization tag in the web.config. We could add additional lines to allow further roles or users. In this case, we simply want to deny everyone else, so notice the deny users * wildcard below.

Web.config

<system.web>
<authentication mode=Windows/>
<identity impersonate=true/>
<authorization>
<allow roles=BUILTIN\Administrators/>
<deny users=*/>
</authorization>
</system.web>

Here’s a handy tip: When testing use the whoami command. This will show you all the groups the logged in user belongs to which is handy when testing.

If you are making modifications to local or domain groups for the current user (probably your own account for testing) ensure that you see the group information you expect with the whoami /groups command. If you have just added yourself to a test group, remember that you must logout and log back in to see these changes.

Now you could stop here, you’re website is secured. However, if you do the user will get IIS’s rather nasty 401 error page. It would be much nicer to show our own custom error or possibly redirect the user to a registration page of some sort. The problem is we’ve restricted the entire website, so even a harmless error page requires authorization. What we need is an exception, which we can do be adding the snippet below the system.web closing tag.

<location path=AccessDenied.aspx>
<system.web>
<authorization>
<allow users=*/>
</authorization>
</system.web>
</location>

What this has done is specify a specific file to have different authorization requirements to the rest of the website.  Optionally we could have specified a directory where we can place CSS, image files, a masterpage, or other resources we may want to allow access to.  In this example, we are only allowing all users to see the AccessDenied.aspx page.

You might think that using the customErrors section in the web.config would be the last step to redirect the user to the AccessDenied.aspx page. However, IIS has other ideas!  IIS catches the 401 before it ever consults with ASP.Net and therefore ignores your customErrors section for a 401. Use the below workaround in your global.asax.cs to catch the 401 and redirect the user to our error page before IIS has a chance to interfere.

Global.asax.cs

protected void Application_EndRequest(Object sender, EventArgs e)
{
if (HttpContext.Current.Response.Status.StartsWith(“401”))
{
HttpContext.Current.Response.ClearContent();
Server.Execute(AccessDenied.aspx);
}
}

If we have further security requirements, we can do further checks any time by getting the IPrincipal from Page.User or HttpContext.Current.User. We could enable a button in our UI with a simple statement such as:

adminButton.Enabled = Page.User.IsInRole(@“domain\domainadmins”);

While the solution seems obvious and elegant, it took a bit of searching and playing to get it to work just right. There are other ways of centrally enforcing security such as Application_AuthenticateRequest in global.asax. However in this case it is far more configurable to take full advantage of the settings available to us in the web.config.