Posted on : By : TCTJr Posted in : New Kewl Stuff

Probablistic Databases For Predictive Content

The Other PDF To You

The Probability Something Will Happen

 

Digital Remote for Your Life

Well folks we are going to shift gears here a little and get back to some hardcore Ideas2Bank discussions concerning technologies.  As of late I have been interested once again in Finding-Not-Searching types of behaviors.  Affinity based systems are once again on the rise.  I will go so far as to say the Age of Affinity is here.  TechCrunch did a writeup recently concerning relevance.  At the end it turned into a pitch for Quora.  However it did have some good ideas concerning continuum of Personalization functionality from complete serendipity to exact personalized context aware information constructs based on geo-location.  I have always been a fan of “lean back” technologies.   Technologies that essentially enable a digital remote for your life.  These types of systems have two common themes: 1) ease of use 2) The probability of usage

Predictive Content and Probability

In today’s world we are trying to create predictive, context aware systems based on the wrong models.  For today’s database architecture: 1) An item either is in the database or is not 2) A tuple either is in the query answer or is not.   This applies to all state-of-the-art data models across the board.  In a probablistic database we have a different construct altogether which is a better fit for the flow of content. For a content prediction event driven system we can assume the events are precise and have no ambiguity. Instead, it is the future event stream that is unknown and the matching of a pattern at some point in the future is predicted with some probability.   When, Where and How are the operatives for this type of predictive event, f(Wh,Wr,H) if you will.  Also note I mentioned the word stream.   I believe given the current and future infrastructures for processing we are bringing back some of the same analogies for large array signal processing frameworks.  The probablistic database models set up extremely well for these types of event processing mechanics.

For a probabilistic database we have:

  • “An item belongs to the database” is a probabilistic event.
  • “A tuple is an answer to a query” is a probabilistic event
  • Can be extended to all data models; we discuss only probabilistic relational data

Probabilistic databases distinguish between the logical data model and the physical representation of the data much like relational databases do in the ANSI-SPARC Architecture. In probabilistic databases this is even more crucial since such databases have to represent very large numbers of possible worlds, often exponential in the size of one world (a classical database).  In complex event processing systems, events from the environment are correlated and aggregated to form higher level events. Uncertainty in the events may be due to a variety of factors including imprecision in the event sensors or generators (eg streams), and corruption of the communication channel possibly dropping events, which can be measured with entropy metrics.  These attributes lend themselves well to fusion systems and the social stream architectures.   Given we are looking at heterogenous data sources that set up for collisions and data source integrity these types of databases hold great promise.    In addition many of these types of database architectures build upon Finite State Machine mechanics for event processing in operating systems.  Of further interest the data is usually imprecise.

Probalistic Databases address types of imprecision whereas:

  • Data is precise, query answers are imprecise
  • User has limited understanding of the data
  • User has limited understanding of the schema
  • User has personal preferences

Notice a “trend” here?  This sets up very well for content flow predictions.  In addition these types of systems hold well for principled semantics for complex queries.  This provides context for the queries where the data is usually imprecise.  Data integration and data hygiene are paramount in social stream systems.  Where data accuracy is important most companies spend 85% of workload cleansing data.  We could use probabilistic information to reason about soundness, completeness, and overlap of sources (think linked data here).  I have listed some of the main sources of research in Probabilistic databases herewith.  As far as I know there are no publicly commercial applications as of yet for this technology.  My bet is we will see some very soon integrated with some of the other NoSQL like technologies.

For a list of current research projects see:

Until Then,

Go Big Or Go Home!

@jaxsoncreole

Posted on : By : TCTJr Posted in : Out In The Fringe

A Couple of Lessons – Responsibility For The Responsible

Responsibility For The Responsible!

Hello all:

I trust this finds everyone well.  We have all the usual worldwide things going on: disasters, wars etc and then in the valley we have huge funding events happening e.g. Color getting $41M. I was thinking ’bout some lessons lived and learned whilst reflecting on past lives and a couple of items came out of the “wet-ware” subconscious: 1) Do The Right Thing 2) Make Decisions Like Your Paycheck Is Irrelevant. (style note: after all these years of Mac programming and working at Apple I still use object style capitalization).  Whilst these item do not appear to be tech worthy or money making – someone said I should teach a Tech Psychology 101 class.

Do The Right Thing

A very successful person in the tech industry who at this point I consider close to iconoclastic said,  “Just Do The Right Thing.”  I am not going to get into the metaphysical aspects of Right/Wrong here suffice to say what this person was making a comment on was have a clear conscience and do not be biased as to the correct technical and business decision.  In the long run you will benefit and you can always look back and know that what you did as a technical professional was well and just.  This was coming during a very complicated and involved several people with agendas.  For those who are not in the tech industry you would be amazed at how emotional coders, architects and program directors are with respect to some of these decisions.  Then I think that nowadays they are very close to rockstars so what not be emotional?  That said when you have a difficult decision to make “Do The Right Thing.”

Make Decisions Like Your PayCheck Is Irrelevant

Once again someone well known in the industry was walking with me at a conference trade show and due to the level of their success and the amount of well tech power they possessed I used this time to ask questions.  I asked them what is consistent in your success?  Without missing a beat this person said, “Work like your paycheck doesn’t matter.”  Of course your probably saying if this person is that successful they can afford it.  Well dear readers they were not talking about money they were talking about reputation.  Money is temporary.  Reputation lasts a long time – possibly until the big sleep.  It took me a while to truly get my head around this because well time is money and money is time.  Yet the more I pondered this statement the more it made sense (and cents).  Time went on and I started espousing some of these tenets.  Eventually I came upon a situation where I knew one of my colleagues was going into a meeting that could change the face of I/O connectors (think USB and FireWire wars) and I told him say what you know and operate like you do not need a paycheck.  He came out of the meeting and said you know I will always operate in that manner.  It completely freed me from figuring out what to do in high pressure situations.

Responsibility For The Responsible!

First and foremost one thing I do wish is I could have met the person who espoused this statement.  I actually came very close when I was in The Valley circa 93′.  Yet that is a story for another time and place.  The two previous items roll up into this tenets.  For those that are changing the world via technology we have the power to create and destroy.  For some this is the game because well they have made millions and billions, just as the artist who can create to destroy which is part of the art.  Yet even though we create technologies that change the very social fabric of humanity we are still dealing with the human form on a daily basis (caveat emptor for better or worse) and as such we have decisions that will live with us. Conflict is a given.  It is going to happen.  Humans adore war.  Many a people in the tech industry have destroyed many an art – books could be written on the amount of code and product that hit the trash and <delete>.  Part and parcel of this is the aspect of doing what your told instead of being responsible and doing the right thing.  Software is the most scalable industry period.  There may never be another industry as scalable (smart grid maybe but not yet).  The multi-cast nature of changing entire groups of people via one application or a widget is phenomenal not to mention the monetary reward.  As I am fond of saying – Idea2Bank as fast as possible yet this comes with some aspect of Responsibility For The Responsible.  There are those that can and those that cannot its a very stratifying industry.

At the end of the day do what you know – say what you think – create the software you know is true.  Software only knows brutal honesty – either it works or it doesnt and you should be a reflection of that creation.

Until Then,

Go Big Or Go Home!

@jaxsoncreole

Posted on : By : TCTJr Posted in : Out In The Fringe

A Couple of Lessons – Responsibility For The Responsible

Responsibility For The Responsible!

Hello all:

I trust this finds everyone well.  We have all the usual worldwide things going on: disasters, wars etc and then in the valley we have huge funding events happening e.g. Color getting $41M. I was thinking ’bout some lessons lived and learned whilst reflecting on past lives and a couple of items came out of the “wet-ware” subconscious: 1) Do The Right Thing 2) Make Decisions Like Your Paycheck Is Irrelevant. (style note: after all these years of Mac programming and working at Apple I still use object style capitalization).  Whilst these item do not appear to be tech worthy or money making – someone said I should teach a Tech Psychology 101 class.

Do The Right Thing

A very successful person in the tech industry who at this point I consider close to iconoclastic said,  “Just Do The Right Thing.”  I am not going to get into the metaphysical aspects of Right/Wrong here suffice to say what this person was making a comment on was have a clear conscience and do not be biased as to the correct technical and business decision.  In the long run you will benefit and you can always look back and know that what you did as a technical professional was well and just.  This was coming during a very complicated and involved several people with agendas.  For those who are not in the tech industry you would be amazed at how emotional coders, architects and program directors are with respect to some of these decisions.  Then I think that nowadays they are very close to rockstars so what not be emotional?  That said when you have a difficult decision to make “Do The Right Thing.”

Make Decisions Like Your PayCheck Is Irrelevant

Once again someone well known in the industry was walking with me at a conference trade show and due to the level of their success and the amount of well tech power they possessed I used this time to ask questions.  I asked them what is consistent in your success?  Without missing a beat this person said, “Work like your paycheck doesn’t matter.”  Of course your probably saying if this person is that successful they can afford it.  Well dear readers they were not talking about money they were talking about reputation.  Money is temporary.  Reputation lasts a long time – possibly until the big sleep.  It took me a while to truly get my head around this because well time is money and money is time.  Yet the more I pondered this statement the more it made sense (and cents).  Time went on and I started espousing some of these tenets.  Eventually I came upon a situation where I knew one of my colleagues was going into a meeting that could change the face of I/O connectors (think USB and FireWire wars) and I told him say what you know and operate like you do not need a paycheck.  He came out of the meeting and said you know I will always operate in that manner.  It completely freed me from figuring out what to do in high pressure situations.

Responsibility For The Responsible!

First and foremost one thing I do wish is I could have met the person who espoused this statement.  I actually came very close when I was in The Valley circa 93′.  Yet that is a story for another time and place.  The two previous items roll up into this tenets.  For those that are changing the world via technology we have the power to create and destroy.  For some this is the game because well they have made millions and billions, just as the artist who can create to destroy which is part of the art.  Yet even though we create technologies that change the very social fabric of humanity we are still dealing with the human form on a daily basis (caveat emptor for better or worse) and as such we have decisions that will live with us. Conflict is a given.  It is going to happen.  Humans adore war.  Many a people in the tech industry have destroyed many an art – books could be written on the amount of code and product that hit the trash and <delete>.  Part and parcel of this is the aspect of doing what your told instead of being responsible and doing the right thing.  Software is the most scalable industry period.  There may never be another industry as scalable (smart grid maybe but not yet).  The multi-cast nature of changing entire groups of people via one application or a widget is phenomenal not to mention the monetary reward.  As I am fond of saying – Idea2Bank as fast as possible yet this comes with some aspect of Responsibility For The Responsible.  There are those that can and those that cannot its a very stratifying industry.

At the end of the day do what you know – say what you think – create the software you know is true.  Software only knows brutal honesty – either it works or it doesnt and you should be a reflection of that creation.

Until Then,

Go Big Or Go Home!

@jaxsoncreole

Posted on : By : TCTJr Posted in : General Discussion

Reprise Again – The First “T” of a StartUp

Recently Mark Suster presented a writeup at TechCrunch here: Whom Should You Hire? Of interest is the following extraction:

“ If you’re doing a great job at continually recruiting and if you have a company ready to hire several people, at some point when you have enough of a pipeline of talented people you need a way to separate them. I have a long-standing mantra, “attitude over aptitude.” This is assuming a raw minimum of MIPS in the candidate. They need to be seriously smart / talented in their field to make the minimum grade.

But within this “minimum acceptable talent level” you still have a wide variance of “employee types.” Let’s be honest – some uber talented people are PITAs. I never hire them. One bad apple spoils things for everybody.

You don’t see it coming. You figure, “sure, they’re a pain but they produce such high quality work I’m willing to put up with them.” Don’t. The last thing you need is some rat bastard fomenting trouble.

They’re the ones who are talking pop at cocktail parties when they’ve had one too many. They’re having private lunches with other employees talking about how they’ve lost faith in your vision.

When you hit internal moments of doubt you need the team members who say, “Guys, we can do this! We’re up against the ropes but we’re not down. Let’s dig in.” You need team members who do that when you’re NOT there.”

Truer words were never spoken.  We haven’t really changed much in our habits via our short time of evolution here on Earth.  Our survival instincts kick in and usually people freak out and do weird stuff.  I could write tomes on the things I have seen people do within times of tech world crisisdom.  There is an old adage “Do Not Panic and You Will Live.”  Most panic and freak out – but those that adapt and hang in there usually win out.  I wrote sometime ago about the The Three “T”s of a StartUp.  I also wrote sometime ago about the companies bidding on talent way before Zuckerburg bid on twitter for teams: Revisiting The Three T’s. I then wrote about the obvious talent shortage and how how we are going to see a swarming affect of teams much like the days of clans in Quake: Mercanaries For Hire.

Yet even with the best of the best of the best.  One thing still stands true: At some point the attitude does over shadow that compiler count.

Go Big Or Go Home.

@jaxsoncreole