TmMq - Trivial MongoDB Message Queue

I've just pushed a very simple message queue system that used MongoDB as the data store. I've found this useful for testing message queuing and projects where I dont want to deploy a full blown message queuing system.

Hopefully it will help someone else too.

You can get the source and binaries from github
             https://github.com/andrevdm/TrivialMongoMessageQueue


Below is the readme from the project
----------------------------------------

TmMq

TmMq - Trivial MongoDB Message Queue is a very simple .net message queuing system built on MongoDB
It is not in any way meant to compete with any of the fully fledged messaging solutions (Hortet, ActiveMQ etc) but it is a nice, lightweight alternative that has proved useful to me.

Features

1. No TmMq server
2. Send & receive
3. Publish / subscribe
4. Redeliver on error with limit on retry
5. Limit on delivery (at-least-once delivery)
6. Message expiry
7. Message holding (only deliver in future)
8. Errors logged in message
9. Dynamic properties collection
10. Synchronous and asynchronous receive
11. Written in C#

 

TODO

1. Triggers based on tailable MongoDB cursor. I'm not sure this is necessary, I will implement it if I find I need it.
2. More unit tests

 

Licence

FreeBSD License. See licence.txt

 

Usage

See the unit tests for examples of all the features including pub/sub, retry, errors etc.

 

Send & receive

using( var send = new TmMqSender( "TestSendBeforeReceiveStarted" ) )
{
     var msg = new TmMqMessage();
     msg.Text = "msg1";
     send.Send( msg );
}

using( var recv = new TmMqReceiver( "TestSendBeforeReceiveStarted" ) )
{
     ITmMqMessage recieved = recv.Receive().FirstOrDefault();
}

 

Pub/sub

using( var rcvr1 = new TmMqPubSubReceiver( "TestPubSub" ) )
using( var rcvr2 = new TmMqPubSubReceiver( "TestPubSub" ) )
using( var rcvr3 = new TmMqPubSubReceiver( "TestPubSub" ) )
using( var rcvr4 = new TmMqPubSubReceiver( "TestPubSub" ) )
{
     var r1 = new List();
     var r2 = new List();
     var r3 = new List();
     var r4 = new List();

     rcvr1.StartReceiving( 1, r1.Add );
     rcvr2.StartReceiving( 1, r2.Add );
     rcvr3.StartReceiving( 1, r3.Add );
     rcvr4.StartReceiving( 1, r4.Add );

     using( var sender = new TmMqPubSubSender( "TestPubSub" ) )
     {
            var msg = new TmMqMessage();
            msg.Text = "ps-" + i;
            sender.Send( msg );
     }

Parameterised queries–don’t use AddWithValue

I’ve just had another run in with the SQL query optimiser. Here is my tale of woe.
I had a very simple parameterised query. Something like this

select * from People where ID10 = @idnumber

However when I looked at the SQL execution plan it looked like this

The thing to notice here is that it is doing an index scan. This made no sense to me since the ID10 column is indexed and so I should be seeing an index seek.

Using SQL profiler confirmed that this query was taking nearly half a second on our production server, which was way too slow.

This is the query as recorded by SQL profiler after being executed by the C# code
exec sp_executesql N'select * from People where ID10=@id', N'@id nvarchar(10)', N'1001010001'

And here is the table

What was strange is that when I executed the SQL without a parameter

  exec sp_executesql N'select id10 from People where ID10=''1001010001'''

I got this execution plan

An index seek, exactly what I wanted. This query took between 1 and 10ms, so more than 400 times faster!

After much searching I finally found the answer;

This parameterised query works perfectly, it uses an index seek

exec sp_executesql N'select id10 from People where ID10=@id', N'@id varchar(10)', '1001010001'

The difference? One letter… This query passes the ID number as a varchar not an nvarchar. Since the index is on an varchar, passing in a nvarchar means that there will be an index scan not a seek. I have no idea why SQL does not first convert to a varchar and then do a scan, but it does not…

The culprit in the C# code was this line
cmd.Parameters.AddWithValue( "id", idNumber )

The AddWithValue forces .net to infer the type you are passing in and since all strings in .net are unicode the parameter is sent as an nvarchar.

Know this, the fix was trivial

cmd.Parameters.Add( "id", SqlDbType.VarChar, 10 ).Value = idNumber,

Here the type and length are specified explicitly so the query is correct and SQL uses an index scan.

So in summary don’t use AddWithValue

BlockingCollection & parallel yields

.Net 4 has many new features. My favourite at the moment are the classes in the System.Collections.Concurrent namespace. System.Collections.Concurrent has four thread safe collections; ConcurrentQueue<>, ConcurrentStack<>, ConcurrentDictionary<> and BlockingCollection<>.
Here are two articles that discuss these collections

• Thread-safe data structures .NET 4.0 (part 1)
• Parallel Programming in .Net 4.0 and Visual Studio 2010: BlockingCollection<T>

I’m finding the BlockingCollection’s GetConsumingEnumerable() method incredibly useful. What it does is return an IEnumerable<> that removes items from the collection and blocks while the collection is empty. This combined with the fact that the BlockingCollection is thread safe makes for very simple code.

As an example image that you have this method that downloads data from several locations (DownloadSourceStreams), parses the data (ParseStream) and yields the processed results (ImportData)

public override IEnumerable ImportData()
{
 foreach( Stream inputStream in DownloadSourceStreams() )
 {
  ParsedData data = ParseStream( inputStream );
  yield return data;
  }
}

public IEnumerable DownloadSourceStreams()
{
 foreach( string url in m_sourceUrls )
 {
  yield return DownloadStreamFrom( url );
 }
}

It would make sense to download all the data in parallel. However you can’t simply make the foreach in ImportData() a Parallel.ForEach since you can't yield from within a Parallel.ForEach. So what you need to do is download the data in parallel and then have a single point for yielding all the results. The BlockingCollection makes this easy

private BlockingCollection m_imported = new BlockingCollection();

public override IEnumerable ImportData()
{
 Task.Factory.StartNew( ParallelImportData );
 return m_imported.GetConsumingEnumerable();
}

private void ParallelImportData()
{
 Parallel.ForEach( DownloadSourceStreams(), inputStream =>
 {
  ParsedData data = ParseStream( inputStream );
  m_imported.Add( data );
  } );
  
  m_imported.CompleteAdding();
}

public IEnumerable DownloadSourceStreams()
{
 foreach( string url in m_sourceUrls )
 {
  yield return DownloadStreamFrom( url );
 }
}

The code now works as follows

• ImportData() starts a new task to import the data (line 5) and then returns the consuming enumerable (line 6)
• ParallelImportData() is then called on a separate thread/task and does the downloads in parallel by using Parallel.Foreach (line 11). Each imported item is then added to the blocking collection (line 14).
• When the import has been completed the BlockingCollection’s CompletedAdding() method is called.

When the consumer calls ImportData() it gets an IEnumerable (consuming enumerable) that blocks while there is no data in the collection. As soon as there is data it iterates over it and removes it from the base collection. This continues until CompletedAdding is called.


What is striking about this code is that all of this happens without any explicit locking, the blocking collection handles it all for you.

.net REPL

I often want to test things quickly in .net without creating a new project. There are quite a few options.

REPL

Firstly the read-eval-print-loop options. These are great for quick tests.

Most of the dynamic languages have a REPL. Personally I use booish because I like the boo programming language. IronPython and IronRuby also have REPL. You can even run IronPython in your browser if you want.

The best C# REPL I’ve seen is gsharp from Mono. You can happily run Mono side by side with Microsoft’s .net frameworks. So there is no reason not to try this.

Compiling

For more complex tests a light weigh editor/compiler are useful. By far my favourite in this category is SciTE. SciTE is an incredible editor that will compile .cs files (and many others) out of the box. Just open a .cs, press F7 to compile and F5 to run. It can be configured to edit pretty much anything. Its light weigh, cross platform and it is free. IMO its an editor that every developer should take a look at.

Other options include Snippet Compiler which has a nice IDE and intellisense. Also take a look at Snippy, and the reflector snippy addin.

C# T-Tree

I have just created a T-Tree project on github: http://github.com/andrevdm/ttree/tree/master

“A T-tree is a balanced index tree data structure optimized for cases where both the index and the actual data are fully kept in memory, just as a B-tree is an index structure optimized for storage on block oriented external storage devices like hard disks. T-trees seek to gain the performance benefits of in-memory tree structures such as AVL trees while avoiding the large storage space overhead which is common to them.” (from wikipedia)

See also: Tobin J. Lehman and Michael J. Carey, A Study of Index Structures for Main Memory Database Management Systems. VLDB 1986 for a comprehensive discussion of T-Trees

The project is a C# implementation of a T-Tree. There is also a very simple command line profiler and a GUI for visualising inserts and deletes.

There are still a few things outstanding (e.g. Making the class enumerable and supporting the visitor pattern) but all the basics (insert, search and delete) are now working.

I'm sure that more can be done to speed up this implementation but it already performs very well. That coupled with the efficient use of memory makes it quite an attractive data structure.

A quick disclaimer: I have only just gotten the basics working I'm sure there are still some bugs lurking. I'll be doing more testing and adding more unit tests as I have time.

The code is released under the BSD licence.

Please let me know if you find it useful, spot any bugs or can think of way to improve it.

Visual Studio Dot Debugger Visualiser

Graphviz is amazing, I've never found a better or easier to use tool for generating graphs and data visualisation. The dot language is very easy to learn and the documentation is very good indeed. If you are doing any form of visualisation its worth taking a look at.

In the past I've used it for visualising compiler ASTs and data structures. (Which reminds me Jim Idle posted a nice ANTLR to dot conversion sample).

At the moment I'm working on a couple of data structures. Visualising them while developing the structures helps a great deal with spotting obvious errors and checking that things look as expected (yes I have unit tests too :). However it was a little inconvenient to have to keep writing the dot out to disk and generating the images manually.

Luckily Visual Studio supports visualisers and they are easy to write too. So I've just created a dot visualiser. My classes now have a ToDot() method which return a string. Whenever I want to visualise them I simply add a watch and select the dot visualiser. It could not be easier.

 

Download

VS 2008 Source code
VS 2008 Binaries

Installing the binaries

1. Copy the binaries to your visualisers directory (%USERPROFILE%\Documents\Visual Studio 2008\Visualizers)
2. Edit the config file and set the path to dot.exe

Disclaimer

I’ve not made much effort to make this visualiser robust. It helps me during debugging, that’s all. 

Update: Thanks Riaan for spotting an horribly embarrassing  bug in the code :).

Species Browser

update 2009/01/01

[Warning! Abandonware: I am no longer developing/maintaining this application]

Introduction

Species Browser is a simple application for browsing information saved from the web. Both a desktop PC version and a Windows Mobile version are available. I wrote it for my own use but I'm releasing it here as you might also find it useful.

Species Browser browses information you have saved. It does not come with any of its own data.

Folders

For it to work you need to save your information one folder per species. In the picture below you can see the folder structure on my mobile phone.
Here you can see my plant folders. I have similar folders for fish and 'Other' which contains info on invertebrates etc. Each of these top level folders is referred to as a collection. You can setup whatever folder structure you want. I then have the exact same folder structure on my desktop machine.

Files

In the pic below you can see the files in one of these folders. The images are used for thumbnails. The saved web pages are displayed on the documents tab and the notes file is displayed on the main page.

---

The Windows Mobile Version

The pic above shows the main screen displaying information on Hygrophilia polysperma.

On the "Docs" tab you can view your saved web pages


The windows mobile version can only view htm and html pages. It can not display mht files. So if you are saving web pages with IE select "Webpage, complete" and not "Web archive".

Setup

Before you can start using the mobile app you need to setup your collections. I.E. you need to tell the app where to look for your files. To do this click on the "Menu", and select "Setup Collections"

You will then see the collection setup page. On this page you can create links to new collections and edit your existing collections.


To add a new collection

1. Click on the "New Collection"
2. Enter a name
3. Click the browse button"..." and browse for the folder containing your collection
   

Selecting a collection

You can now select a collection from the main menu

Selecting a species

There are two ways to select a species to view. Either select the item from the drop down or click on the "Species" menu item.
If you click on the "Species" menu item you will see this screen



On this screen you can select an item from the listbox or search for items containing a phrase. To search for a phrase type in the edit box at the top of the screen.

Selecting a document

Once you have selected an item from the listbox click the "Select" menu item to load it.

Once you have loaded a species you can view your saved documents by clicking on the "Docs" tab. As with the species page you can either select a document from the drop down or search for it.

Keyboard shortcuts

This pics shows how you can navigate the mobile app without using a stylus

---

The Windows Desktop Version

I've done a lot less work on the desktop version as I mainly use the mobile version. However it does work...


The PC version is very similar to the mobile version the only difference being that the PC version allows you to edit the notes.

Setup

Click on the "Tools" menu, select "Collection Setup"


Click on "Add" to add a collection. The browse button lets you browse for your collection folder

---

Download

Both versions are free. I may update them from time to time but I'm not making any promises :)


Prerequisites
To run the windows mobile version you will need to have the Microsoft .Net Compact Framework version 2 installed. If you don't have it you can download it here.

To run the windows desktop version you will need to have the Microsoft .Net Framework version 2 (this is not the same framework as the one above) installed. If you don't have it you can download it here.


Species Browser for Windows Mobile version 1.0.2.3
Download the mobile version's installer here

Species Browser for Windows Desktop machines version 1.0.2.1
Download the desktop version's installer here

---

Feedback

Let me know if you are using Species Browser by emailing me <dart-software@pobox.com>.