How to create Self learning data product

2

I am trying to build price recommendation solution for clients in a scalable manner. I have two choices as below.

Professional service: Statistician involvement to build regression model or any other kind of predictive model that fits specifically to client data and can be used.

Issue: So on the long run there will be issues around scalability as one analyst cannot build model simultaneously for hundreds of clients who want to come on board and use this service. Hiring 1 analyst for 100 clients is not an option for me as i do not run this operation.

Product: Plug and play with least human intervention. i.e. Some kind of self learning/artificial algorithm that works for any and every client. Additionally, it adjusts to new trends that appear over time which might not have been captured in the existing model. Personally, i will be biased towards this kind of solution.

Issue: This is the ideal solution, however i am not sure how to build this and how to approach in this direction.

To give context, a client in Europe might have same data structure in their sql database in terms of name, variable type in comparison to sql database of a client who is based in Singapore. However, Singapore data might have different trend, seasonality and pattern in comparison to data in Europe. Is there any way that a single Machine Learning or Artificial Intelligence or Deep Learning or Neural Net algorithm can be built and used for all regions and clients without having to update that model manually for any new client?

My background: i have previously worked in a professional service landscape and have statistics/predictive modelling background.

StatguyUser

Posted 2015-12-12T13:15:44.547

Reputation: 825

Does the area you want to create a product in (price recommendation) usually have a highly regular set of input data across all kinds of clients? I would guess not, which would make creating a single one-fits-all product very hard. You could create a product around the technical model, but there would still be a lot of work adapting to specifics of each client. How well it could scale depends critically on how much qualitative difference there will be between clients' data. – Neil Slater – 2015-12-12T22:55:22.783

How do you define 'regular' when you say regular set of input data? Also, what did you mean by "technical model"? – StatguyUser – 2015-12-13T03:02:53.470

Probably I should say "standardised" instead of "regular". I mean arranged in a very similar way, having an identical ontology, so that the same process can be used to extract features each time. By "technical model", I mean the code that generates a predictive model from the input - often the easy bit once you have arranged, understood and cleaned the data. – Neil Slater – 2015-12-13T08:37:27.780

Thanks for your help on this. So i assume that answer to my original question is option1, which is every time new client on boards, that statistical model will have to be built from scratch so that mathematical equation can be developed and then it can be deployed. although technical model may remain same. If we are not going off topic, i find some companies write this in their site, which is misleading. – StatguyUser – 2015-12-13T10:12:41.313

Actually I do not know the answer. It depends on your answers to my question - for the price recommendation problem that you are looking at, how "standardised" can you expect client data to be for your service? If the answer to that is "not at all", then it is not possible to fully automate it. If the answer is "most clients can and will arrange the data to have very similar input structure and fields", then you have a good chance. – Neil Slater – 2015-12-13T10:23:59.310

Ok, to give context, a client in Europe might have same data structure in their sql database in terms of name, variable type in comparison to sql database of a client who is based in Singapore. However, Singapore data might have different trend, seasonality and pattern in comparison to data in Europe. Is there any way that a single Machine Learning or Artificial Intelligence or Deep Learning or Neural Net algorithm can be built and used for all regions and clients without having to update that model manually for any new client? – StatguyUser – 2015-12-13T13:09:54.130

That seems more tractable. I don't know the answer, but I suggest you put that last comment of yours into the question, because it may help for someone who knows your problem domain better. – Neil Slater – 2015-12-13T14:42:25.840

I suppose it's technically possible, but I doubt that it would give you good/efficient learning. Multinational corporations that invest in this sort of thing, like Walmart, don't rely on automating price determination because there are extensive time lags and human considerations that can't be put into a model. Product recommendation engines work because there is a constant and reliable stream of data of what customers click on to purchase. The same is not true for all the factors that determine a product's price. – rocinante – 2015-12-13T23:39:54.433

No answers