Are you ready for big data and big challenges?

The information age is exploding all over us and it’s a big data big challenge.  Recently I made the decision to migrate from my hybrid operating system and productivity in my lab environment to a new native solution, with all my systems using the same operating system.  I had exceeded the number of contacts that my existing hybrid solution could support.  In the migration to my new native OS system I found numerous challenges over a two month period that were frustrating and difficult to solve.  I found this process to be, on a small scale, just like that of enterprise global networks with big data challenges of a similar nature.

My first challenge was finding a subject matter expert (SME) that could actually answer the specific tactical questions I had to ensure that my possible solution to the problem and needs would work as desired.  I spent hours on calls, researching the Internet, and asking others that I knew for support but alas, finding that one person that actually got it – had authoritative experience – eluded me as I started the journey of moving to a new OS and productivity solution, etc.  I ended up making a purchase based upon a variety of calls and talks but found that when it arrived it clearly was not what I had discussed; wrong parts were ordered and the new laptop wouldn’t work for my demanding needs.  Upon returning and escalating to management I was able to finally talk to a senior level SME that ‘got it’ – we had a mind meld on hardware, discussed a few options – and came up with a solution that does work and fixed the order.

A few weeks later I found my order was delayed.  It’s a heavily customized based solution that had to be built and shipped overseas and a variety of other production based dependencies ended up delaying it around 6 weeks.  Meanwhile I found that there were operational challenges on other former parts and issues that had to all be resolved, one by one, which didn’t help in the happy factor area of my life but eventually resolved.  Once the hardware showed up it was everything I thought it would be and I was pleased and excited.  Lesson learned – big data projects and new ventures rarely happen on time as expected – it’s new territory.  I had a contingency plan in place for my production so I was fine during this time frame.

Data migration is always a huge topic – this is why I most often prioritize any of my big data projects on my cloud/large scale storage/redundancy and database components and needs first.  It has to be reliable, accessible, scalable, affordable, and redundant.  My solution is the same and I soon found it was incredibly complicated.  I tried to use non-native productivity software – because it was something I had used for a long time, and found that it just won’t work.  In fact, after analyzing the protocols and other design components I realized it was designed to not work for competitive disadvantage.  I coughed up the dough to purchase the software to aid in legacy management of older documents as well as new construction and editing of documents, but it was clearly not going to be my primary productivity based service going forth for my new normal ops.

I have so many TB, yes, TB’s of data, that I had to make some hard choices.  I went through tons of old data and deleted, archived, etc, but still ended up with a massive amount of data on my two NAS devices as well as access to a large quantity of data for my primary production machine.  Movement of that data through my network was interesting as I was starting to tap the limits of my data storage.  I’ll have to make some strategic decisions soon on what I choose to archive, and for how long, despite my 24 TBs of data storage currently in place.  Meanwhile I found a solution to systematically stream data through the network to efficiently migrate primary production data to my main new solution.  This went really well and new data storage is fairly cheap which is an age old truth in the world of big data.  Yes, I am a fan of the cloud for the right resources for the right reasons with the right provider, but that was not required in this situation.

Migration of my productivity, specifically my calendar and my contacts, was interesting.  I learned new formats and protocols and various specs to figure out how to accomplish my goal in the most efficient and effective manner. I couldn’t find any experienced folks that could give me good answers – frustrating to have so little experienced talent in our security industry and that will only get worse.  I got the majority of my calendar items moved over without much trouble but found that the new interface and alerting options were not nearly as good as former solutions.  That’s been killer for me as I’ve been missing meetings and not getting notifications in a way that I’m used to for this entire century.  You can teach an old dog new tricks but it involves a lot of meetings and time 🙂

Migration of contacts proved to be much more challenging.  This was the original need for migration to the new native solution.  I had so many that a CSV dump contained almost a half million data points.  When I attempted to import the data I found after about a day that it ran out of memory and could not proceed despite having a beefed up system.  It was simply too much to consume and process.  I ended up studying the formats, trying small tests, to identify the best methods of import and transfer, parsing, etc.  I was able to finally get a solution that worked to successfully important and update all my contacts.  Finally!

I still have some ironing out to do on my day to day operations and notifications but I’ll get there.  I’m glad that I intentionally used a single system for a long time but then, at the right time, chose to move in a new long term direction to meet my new needs.  It involves commitment and lots of challenges along the road, but with persistence, success can be found.  It also takes extreme focus and critical thinking.  We have a lot of big data challenges ahead of us in this era with few people to help lead us in that direction.  Plan carefully, glean what you can from others, and work small projects before working larger ones if at all possible.  Always have the long term strategic view in place as you execute upon the immediate tactical actions required to move forward.