Ismail Seyfi's Blog

I recently had a chance to introduce MongoDB in one of my projects. This application was a data collection system. Its main purpose was to be able to ingest large amounts of data from different data sources. These sources had different formats for data. Some were in XML, some were in CSV and some were in binary. They all had one thing in common though and that was the large size of data. The core of the application was written in a multi-threaded .NET application that was using SqlBulkCopy with a SQL Sever 2012 addition. SqlBulkCopy is one of the preferred (optimized) ways of loading data into SQL Server. The goal of the application was to be able to parse any given file with its given format definition/configuration file. Therefore, it needed to be able to store data into SQL Server that it never stored before. This provided challenges since Relational Data Management Systems require you to define your data structure before you can store data (i.e you must create your tables). Ther...

I recently ran into an interesting situation with SQL Server specifically SQL Server 2012 that involves "Lock Escalation" that usually resulted in dead-lock scenarios. My application is responsible for parsing very large binary/text files and inserting data into SQL tables. My original thought was to handle this large dataset insertion in a single transaction so that data consistency would be achieved. The application has two major components. One of the components is a web application that accesses some of the tables that are used in the bulk insertion I mentioned above for basic verification and validation. When I was running some test cases, I noticed that my application became very un-responsive and it would eventually time out when the time-out values were reached. SQL Server provides some system tables that maintains information about the locks it keeps and any suspended threads due to these locks. You can use some of the following queries below to query your maste...