Career has been a constant struggle

[NickK]

Fear of the unknown can be a big issue too along with perceived loss of status.

Unfortunately people won't interview you for something new over 40. Your expected to be able to make your own way (ie start a company) it appears.

OP - I would STRONGLY recommend you look at quantum.

Here goes - I spent 6 months running a programme to provide recommendations to HSBC's group chief-of-staff following a hackathon entry and demonstration of two quantum proof-of-concepts for financial risk analytics. The recommendations covered risk, data science and security. I then spent almost 1.5 years selling quantum cybersecurity within a company that did a lot of quantum research and work across many industries.

Statistics and quantum aren't far apart. Most of the quantum systems are cloud based and you can use a simple language to play or code up concepts. This is includes simply playing with Bernoulli variables etc, through to full on optimisation and beyond. a lot of toolkits have non-quantum computer based simulators available that you can run on your PC/Mac etc (under the hood they use markov chains etc).

The big thing with "data analyst" in large organisations is it quickly becomes "data cleanser" of legacy and general tripe. If you want to hit the maths properly then you're looking at statistician jobs like Quants or modelling.

I remember being at in a a room with the Group risk analytics director with a load of CIOs, and two individuals with statistical analysis and modelling knowledge (one from Higgs boson research background and one with quantum spin and number probability). Every CIO in that room (HSBC had 350 C-level executives and 380,000 people at this point) had a PHD in maths related to quantum research or superconductors etc.

Quantum is not really a faster data processor - more of a how to build a better data processor. This specific fact is what is driving a lot of competitive work and opportunities for physics and maths grads at this present time.

I would say over the next 5-10 years the quantum piece for new markets will come online as businesses understand how they can use quantum to open new opportunities and not just optimise existing ones. This is ignoring the quantum cybersecurity armageddon.

 

[Career Compromise]

There's a lot to unpack here.

My background btw, is 10 years doing what used to be called Business Intelligence, building data warehouses mostly, for telcos, retail, wholesale, and then mainly investment banks. Got bored of it though and have been in the games industry the last 10 years.

Your interest.....sports, transport & logistics, anything geographical, mortgages, retail? Seems pretty broad. The most interesting analytics work I did was some new reports for Pret A Manger that broke down their wastage by product and store on a heatmap...it was pretty obvious which stores/products where the problem, and it immediately saved them some big sums of money. I can't say I really care about sandwiches though....does the actual industry matter?

If what you are really interested in is the actual data analysis, then you need to know your SQL/MDX, or whatever the tech is these days for querying relation and multi-dimensional databases. A lot of places will be using python/ML for getting insight from their data, but I suspect that majority of places are getting most of their useful data from plain old database queries still.

VBA macros are junk old tech and while some old places probably still use them to hold their reporting systems together with bits of string, that's not where the money or work is. It'll be in databases with an abstraction layer on top like Business Objects/Microstrategy (I may be showing how long I've been away from that industry here).

You're Pret example reflects what I'd like to do in retail i.e. try to have enough sandwiches in stock for people to buy but not too many that they have to get thrown away.

The issue I have with the industry I'm in is that it's not really needed, it's quite a basic concept with needless complication to make it sound like something it's not. It's also full of 25 year old directors of nothing useful with a degree in the history of art who think they know more about data than someone with a data job.

Regarding tools and technologies, in the beginning everything was in Excel so I had no choice but to use VBA. As time went on it became more SQL, Python etc but purely from an enjoyment perspective I prefer VBA. I've not used VBA for a while now though.

If you’re interested in mortgages OP, there are literally tons of data analyst roles at banks. There’s always loads of demand. Shouldn’t be a problem to find something else.

I did have interviews with banks during my unemployment period.

Almost certainly an issue with your interviewing ability then and perhaps a need to brush up on technical skills. You've got the right degree for it at least, plenty of "data analyst" types are more like developers who have rebranded, they can learn the relevant tools but often don't really have much understanding of statistics. Then again if the employer just requires someone to do some SQL monkey work or make pretty reports in Tableau or PowerBi then meh...

All the areas you've mentioned should have plenty of demand for a maths/stats grad who code, Python is quite popular thanks to things like Pandas, NumPy, PyTorch, Scikit-learn etc.. but you could be in a role using R too. The problem is "data analyst" can be quite a broad term ranging from roles that might otherwise be labeled "data science" through to the aforementioned monkey work.

Re: sports - there are some big gamblers/betting syndicates who employ statisticians, in particular re: football betting - AFAIK the pay can be very good and (AFAIK) might include participation in the gambling syndicate too.

You might still have some use for Excel/VBA, especially in banks, though you don't really have the domain knowledge/relevant experience it seems (maybe worth a shot tho...) - arguably if something can be done in VBA without too much faff then it's much easier to give a trader a spreadsheet that they can simply use right away than have them muck about with DLLs etc..

I've dug out the feedback from the interviews I had for the job I wanted the most during my unemployment period. First interview went well, second interview a disaster.

First interview:

"Candidate demonstrated a good understanding of many of the required tools and skills of a data analyst, and showed a strong awareness of some of the challenges and responsibilities that come with a senior role. Some of their answers to the technical questions (on SQL, Python, web analytics) could have gone into more detail on syntax, libraries and concepts they have used in the past, and this would have made it easier for me to gauge their level of understanding, but what they did provide satisfied me that they have what it takes to do the job (and the ability to fill in any gaps with a bit of training). They gave good examples of their mentoring and leadership experience within a data capability, which I would hope to see them replicate within this company."

Second interview:

"Candidate was good talking about the high level concepts and approaches, but when drilling into details they softened struggled to articulate their answers or did not have the awareness. As a data analyst I would expect them to have a much stronger grasp of SQL and they struggled with some basic concepts.

Candidate struggled to give examples of requirements gathering, which is a key aspect of interacting with customers to understand the needs the analysis or visualisations. Again this is a key area

Most of the examples candidate used were about creating VB macros in Excel. This did not sell their potential very well, whereas their CV shows more modern technologies and approaches."

 

[mid_gen]

You're Pret example reflects what I'd like to do in retail i.e. try to have enough sandwiches in stock for people to buy but not too many that they have to get thrown away.

The issue I have with the industry I'm in is that it's not really needed, it's quite a basic concept with needless complication to make it sound like something it's not. It's also full of 25 year old directors of nothing useful with a degree in the history of art who think they know more about data than someone with a data job.

Regarding tools and technologies, in the beginning everything was in Excel so I had no choice but to use VBA. As time went on it became more SQL, Python etc but purely from an enjoyment perspective I prefer VBA. I've not used VBA for a while now though.

I did have interviews with banks during my unemployment period.

I've dug out the feedback from the interviews I had for the job I wanted the most during my unemployment period. First interview went well, second interview a disaster.

First interview:

"Candidate demonstrated a good understanding of many of the required tools and skills of a data analyst, and showed a strong awareness of some of the challenges and responsibilities that come with a senior role. Some of their answers to the technical questions (on SQL, Python, web analytics) could have gone into more detail on syntax, libraries and concepts they have used in the past, and this would have made it easier for me to gauge their level of understanding, but what they did provide satisfied me that they have what it takes to do the job (and the ability to fill in any gaps with a bit of training). They gave good examples of their mentoring and leadership experience within a data capability, which I would hope to see them replicate within this company."

Second interview:

"Candidate was good talking about the high level concepts and approaches, but when drilling into details they softened struggled to articulate their answers or did not have the awareness. As a data analyst I would expect them to have a much stronger grasp of SQL and they struggled with some basic concepts.

Candidate struggled to give examples of requirements gathering, which is a key aspect of interacting with customers to understand the needs the analysis or visualisations. Again this is a key area

Most of the examples candidate used were about creating VB macros in Excel. This did not sell their potential very well, whereas their CV shows more modern technologies and approaches."

Your interview feedback confirms what I've said, Excel and VBA is just not the way any more (it's frightening what stuff at banks was run with it in the past though!).

If you aren't hugely interested in the actual data manipulation side of it, SQL etc, more the VBA scripting....have you considered getting into programming?

As I mentioned, I did data analysis/BI for many years, now I'm an AI programmer in the games industry. Best move I ever made! Lots of the technical stuff carries over.

If you do want to stick with data analyst role....the one thing that will make you stand out above other candidates, is your ability to talk to and understand the business. A big part of the job is just looking at their processes and identify what data from their systems is going to be useful, and why. It's more of a management consultant role if you're good at it. If you can do that part, AND then build the data model/reporting infrastructure.....you'll be laughing.

 

[NickK]

Your interview feedback confirms what I've said, Excel and VBA is just not the way any more (it's frightening what stuff at banks was run with it in the past though!).

If you aren't hugely interested in the actual data manipulation side of it, SQL etc, more the VBA scripting....have you considered getting into programming?

As I mentioned, I did data analysis/BI for many years, now I'm an AI programmer in the games industry. Best move I ever made! Lots of the technical stuff carries over.

If you do want to stick with data analyst role....the one thing that will make you stand out above other candidates, is your ability to talk to and understand the business. A big part of the job is just looking at their processes and identify what data from their systems is going to be useful, and why. It's more of a management consultant role if you're good at it. If you can do that part, AND then build the data model/reporting infrastructure.....you'll be laughing.

True point about banks - VBA doesn't cope with 180GB of realtime data a day, with history, to be actioned within a second or two! Also the skillsets are transferrable across many different areas. Looking for patterns in data is pretty common within most industries.

Most banks are large enough to use multiple cloud vendors to reduce risk, hence you may find transactional processing on say AWS but then data science on Google, and others used for their specific purposes. The good news is that the concepts are typically available on both, also you'd be working in a cross functional team of developers etc that would sort out performance, security etc.

If you want to stick close to the market you're in I would look at Google and AWS and how a data science processing pipeline can be built.

Another option on the front end - specifically customer interactions and optimisations of journeys - most retail/commercial banks will utilise packages like PEGA etc to gleam information from their customer interactions/customer base across mobile, web and general transactions.

TL;DR - my point is your statistics skills are transferrable but you may need todo some work around implementation.

 

[Career Compromise]

Your interview feedback confirms what I've said, Excel and VBA is just not the way any more (it's frightening what stuff at banks was run with it in the past though!).

If you aren't hugely interested in the actual data manipulation side of it, SQL etc, more the VBA scripting....have you considered getting into programming?

As I mentioned, I did data analysis/BI for many years, now I'm an AI programmer in the games industry. Best move I ever made! Lots of the technical stuff carries over.

If you do want to stick with data analyst role....the one thing that will make you stand out above other candidates, is your ability to talk to and understand the business. A big part of the job is just looking at their processes and identify what data from their systems is going to be useful, and why. It's more of a management consultant role if you're good at it. If you can do that part, AND then build the data model/reporting infrastructure.....you'll be laughing.

Programming doesn't interest me. I think the reason I liked VBA above everything else is because it's not proper programming. The more I went down the technical route the more I realised I'd much rather be interpreting the data than transform it etc. I mainly use SQL now, but the key thing is providing actionable insights which is what I want to do. It got very boring very quickly though because there really isn't anything interesting in the data due to the nature of the industry.

I was living off past glories in the final 6 years at my last company. All the things I did that ended with positive outcomes through me being proactive came in the first 4 years. As all the data sat on spreadsheets using VBA almost always featured.

I learned the other technologies like Python in the final 6 years but never really did anything useful thanks to politics and nightmare managers so anything I did using Python was basically someone saying write a Python script to transform the data from this to that. Any questions about what it was to be used for and whether there's a better way was met with not our problem. I was not allowed to speak to people outside the department about doing work for them without a project manager or one of the senior managers with me. I could only do what was asked, thinking about the bigger picture wasn't our problem. It was a truly awful place to work in those last 6 years.

Hopefully I can find some solid examples of good work in my job now, but I still can't help but think I'll be ridiculed for them.

True point about banks - VBA doesn't cope with 180GB of realtime data a day, with history, to be actioned within a second or two! Also the skillsets are transferrable across many different areas. Looking for patterns in data is pretty common within most industries.

Most banks are large enough to use multiple cloud vendors to reduce risk, hence you may find transactional processing on say AWS but then data science on Google, and others used for their specific purposes. The good news is that the concepts are typically available on both, also you'd be working in a cross functional team of developers etc that would sort out performance, security etc.

If you want to stick close to the market you're in I would look at Google and AWS and how a data science processing pipeline can be built.

Another option on the front end - specifically customer interactions and optimisations of journeys - most retail/commercial banks will utilise packages like PEGA etc to gleam information from their customer interactions/customer base across mobile, web and general transactions.

TL;DR - my point is your statistics skills are transferrable but you may need todo some work around implementation.

I've used both Google and AWS. I think the looking for patterns in data is transferable. If I had no interest in sport I could still easily spot that a tennis match had high viewership in Switzerland by looking at the data, but because I am interested in Sport I could see that Roger Federer was playing which is what explains it. It's the latter that's the interesting bit and what my industry lacks.

 

[NickK]

I've used both Google and AWS. I think the looking for patterns in data is transferable. If I had no interest in sport I could still easily spot that a tennis match had high viewership in Switzerland by looking at the data, but because I am interested in Sport I could see that Roger Federer was playing which is what explains it. It's the latter that's the interesting bit and what my industry lacks.

One area in sport is the analysis of players - both individual movement and player strategy/tactic in game play. I remember applying years ago to a company that produced athelete/team analysis software. Allowing coaches to improve performance.

What would be good is a go-pro underwater & above water analysis for triathlons in static swimming pools. People pay 20-30K for a static pool and then train but don't use software to improve their abilities. A 2-3K system, with an ongoing operational subscription after a year would be taken up by the ultra competitive mindset.

 

[dowie]

Programming doesn't interest me. I think the reason I liked VBA above everything else is because it's not proper programming.

Neither is Python in so far as how it is typically used in data science/data analysis.

Python is basically used as a kinda "glue" language in data science, Python itself obviously is a legit programming language despite some incorrectly referring to it as just a scripting language, no shade to the python programmers here however regular python is slow and most of what you're actually using in stats/ML is not Python, it's libraries written using Fortran, C, C++, CUDA etc..

You access these via Pandas, NumPy, SciPy, scikit-learn, PyTorch, TensorFlow etc..

Real programmers have often hate stuff like MATLAB and R but if you're doing stats/ML work then R or MATLAB or also nowadays Python(thanks to NumPy etc..) are standard and unlike with normal programming a key thing in a data science role, if you're implementing something from scratch from say a recent conference paper/new research etc.. is being able to vectorize your code/avoid loops where possible; essentially making sure you're doing as little processing in the native R, MATLAB or Python as is feasibly possible. The loops still exist, but in the far faster languages the numerical/scientific computing libraries are written in. Of course, you might not even need to do that, your role might just require you to use stuff that has already been implemented and can simply be called from scikit-learn etc..

You can also go further still - in ML Engineer roles in the research arms of big tech firms or in deep tech startups then it might be necessary to be familiar with say CUDA etc..

Alternatively, if you want to completely avoid anything like that at all - maybe take a look at some of the BI tools - stuff like Tableau, or Power BI. There is material out there so you can teach yourself this stuff, maybe get a certificate or whatever and go down that route.

Someone above mentioned the soft skills part of things, this is important too - perhaps get yourself a book on business analysis - that's the main job role that traditionally involves talking to the business/stakeholders etc.. probably good to at least be knowledgeable about BA tools/frameworks as there can be overlap with data analyst/BI people there. Of course, a book can't teach you the more generic soft skills, stuff like toastmasters as mentioned earlier could help with that or things like acting or improvisation classes.

 

[NickK]

I’d concur with Python-R-matlab.

you could train with Octave (it’s single threaded) as a free starting point. It also works with python and Jupyter Notepad.

 

[OspreyO]

Unfortunately people won't interview you for something new over 40. Your expected to be able to make your own way (ie start a company) it appears.

OP - I would STRONGLY recommend you look at quantum.

Here goes - I spent 6 months running a programme to provide recommendations to HSBC's group chief-of-staff following a hackathon entry and demonstration of two quantum proof-of-concepts for financial risk analytics. The recommendations covered risk, data science and security. I then spent almost 1.5 years selling quantum cybersecurity within a company that did a lot of quantum research and work across many industries.

Statistics and quantum aren't far apart. Most of the quantum systems are cloud based and you can use a simple language to play or code up concepts. This is includes simply playing with Bernoulli variables etc, through to full on optimisation and beyond. a lot of toolkits have non-quantum computer based simulators available that you can run on your PC/Mac etc (under the hood they use markov chains etc).

The big thing with "data analyst" in large organisations is it quickly becomes "data cleanser" of legacy and general tripe. If you want to hit the maths properly then you're looking at statistician jobs like Quants or modelling.

I remember being at in a a room with the Group risk analytics director with a load of CIOs, and two individuals with statistical analysis and modelling knowledge (one from Higgs boson research background and one with quantum spin and number probability). Every CIO in that room (HSBC had 350 C-level executives and 380,000 people at this point) had a PHD in maths related to quantum research or superconductors etc.

Quantum is not really a faster data processor - more of a how to build a better data processor. This specific fact is what is driving a lot of competitive work and opportunities for physics and maths grads at this present time.

I would say over the next 5-10 years the quantum piece for new markets will come online as businesses understand how they can use quantum to open new opportunities and not just optimise existing ones. This is ignoring the quantum cybersecurity armageddon.

How do you even start in Quantum, especially if you are 40+ ?

 

[OspreyO]

Programming doesn't interest me. I think the reason I liked VBA above everything else is because it's not proper programming. The more I went down the technical route the more I realised I'd much rather be interpreting the data than transform it etc. I mainly use SQL now, but the key thing is providing actionable insights which is what I want to do. It got very boring very quickly though because there really isn't anything interesting in the data due to the nature of the industry.

I was living off past glories in the final 6 years at my last company. All the things I did that ended with positive outcomes through me being proactive came in the first 4 years. As all the data sat on spreadsheets using VBA almost always featured.

I learned the other technologies like Python in the final 6 years but never really did anything useful thanks to politics and nightmare managers so anything I did using Python was basically someone saying write a Python script to transform the data from this to that. Any questions about what it was to be used for and whether there's a better way was met with not our problem. I was not allowed to speak to people outside the department about doing work for them without a project manager or one of the senior managers with me. I could only do what was asked, thinking about the bigger picture wasn't our problem. It was a truly awful place to work in those last 6 years.

Hopefully I can find some solid examples of good work in my job now, but I still can't help but think I'll be ridiculed for them.



I've used both Google and AWS. I think the looking for patterns in data is transferable. If I had no interest in sport I could still easily spot that a tennis match had high viewership in Switzerland by looking at the data, but because I am interested in Sport I could see that Roger Federer was playing which is what explains it. It's the latter that's the interesting bit and what my industry lacks.

Sounds like my place heavily segmented (silo'd) and getting worse. Now everyone comes in with specialised qualification and they won't interview anyone with out it.

 

[OspreyO]

True point about banks - VBA doesn't cope with 180GB of realtime data a day, with history, to be actioned within a second or two! Also the skillsets are transferrable across many different areas. Looking for patterns in data is pretty common within most industries.

Most banks are large enough to use multiple cloud vendors to reduce risk, hence you may find transactional processing on say AWS but then data science on Google, and others used for their specific purposes. The good news is that the concepts are typically available on both, also you'd be working in a cross functional team of developers etc that would sort out performance, security etc.

If you want to stick close to the market you're in I would look at Google and AWS and how a data science processing pipeline can be built.

Another option on the front end - specifically customer interactions and optimisations of journeys - most retail/commercial banks will utilise packages like PEGA etc to gleam information from their customer interactions/customer base across mobile, web and general transactions.

TL;DR - my point is your statistics skills are transferrable but you may need todo some work around implementation.

What sort of places have a need of statistically analysis of 180GB real time data?

 

[NickK]

What sort of places have a need of statistically analysis of 180GB real time data?

financial fraud alerting for customers.

 

[OspreyO]

Programming doesn't interest me. I think the reason I liked VBA above everything else is because it's not proper programming. The more I went down the technical route the more I realised I'd much rather be interpreting the data than transform it etc. I mainly use SQL now, but the key thing is providing actionable insights which is what I want to do. It got very boring very quickly though because there really isn't anything interesting in the data due to the nature of the industry.

I was living off past glories in the final 6 years at my last company. All the things I did that ended with positive outcomes through me being proactive came in the first 4 years. As all the data sat on spreadsheets using VBA almost always featured.

I learned the other technologies like Python in the final 6 years but never really did anything useful thanks to politics and nightmare managers so anything I did using Python was basically someone saying write a Python script to transform the data from this to that. Any questions about what it was to be used for and whether there's a better way was met with not our problem. I was not allowed to speak to people outside the department about doing work for them without a project manager or one of the senior managers with me. I could only do what was asked, thinking about the bigger picture wasn't our problem. It was a truly awful place to work in those last 6 years.

Hopefully I can find some solid examples of good work in my job now, but I still can't help but think I'll be ridiculed for them.



I've used both Google and AWS. I think the looking for patterns in data is transferable. If I had no interest in sport I could still easily spot that a tennis match had high viewership in Switzerland by looking at the data, but because I am interested in Sport I could see that Roger Federer was playing which is what explains it. It's the latter that's the interesting bit and what my industry lacks.

Well you can do "proper" programming in VBA/VB but most places don't. You can access it through other "proper" frameworks.
The issue is, even though MS still support it, they stopped development a long time ago. Even though there is nothing really to replace it.
The other issue is any experience in it, is no longer valued, hasn't been valued for a long time. I still think its powerful and I think many places will struggle to replace it. But that ship has sailed.

 

[OspreyO]

financial fraud alerting for customers.

And is that something anyone would be trying to do in VBA?

 

[NickK]

And is that something anyone would be trying to do in VBA?

my point was that datasets in general are now far larger and complex.

Large focus is placed on optimising the analysis itself and that’s where a large amount of quantum research is currently.

 

[OspreyO]

I'm just trying to make the mental leap from someone working with as the OP did into Quantum, and the disparity of skillsets. Is Quantum heavy in maths far more than general programming would be.

 

[dowie]

I'm just trying to make the mental leap from someone working with as the OP did into Quantum, and the disparity of skillsets. Is Quantum heavy in maths far more than general programming would be.

OP isn't a programmer though and he's got a maths degree.

 

[mid_gen]

What sort of places have a need of statistically analysis of 180GB real time data?

Telcos
Banks
Large retail
Any and all internet-based companies, platforms, service providers, game devs, etc.

In the old days you'd have a data warehouse chew all your data overnight and spit out reports to peruse in the morning. Now you can just have all your systems feeding straight in and getting crunched in realtime, it's just what people expect.

 

[OspreyO]

What's sort of hardware would be need to handle that in real time, or develop against it.

 

[NickK]

What's sort of hardware would be need to handle that in real time, or develop against it.

have a look at this: