I am fairly accustomed to using different methods to solve problems. In most cases I need to break down the problem to find a creative and custom solution because no documentation of something similar can be found. I almost always start by having a simple solution mapped out in excel and then to complete the calculations I create the calculations in code. This is often necessary since the complexity of the calculations become unwieldly in excel.

Installing python packages can be frustrating when using libraries that are not as widely used. It seems the development of those libraries is often more of a hobby and sometimes it expands into something more useful and the installation of those packages is a bit unwieldly.

So there are times when you just need a place to store some notes and I keep losing these notes. I am done searching for them; making copies to send to others; and adding extra descriptions. This page has all the key information that a person needs to customize bas VSCode features.

When I first started working with publicly accessible datasets, I would typically save them to an external hard drive. This lasted about six months before I realized that I had over 200GB of data. It may sound fine but my documentation of where the datasets orinated was not very good and often information about the encoding of the individual features was unclear. Without this information, I would have been just as good off by creating synthetic data of my own.

So I was recently sent about 60 GB of data on a harddrive. The contents were contained in a single file of comma separated values and I was warned that that the data would probably be a bit messy but no one had looked at the data yet. If I had some time, I was requested to browse over the proposed project and give feedback about what would be needed to accomplish the deliverables and what changes to the proposal might be needed if we were to accept this project.

I am a a big fan of the use of Mac Terminal or Windows GitBash when working with git. You will just understand what is happening better when using a command line system. I will conceed that the terminal is not pretty looking like some of the git IDE programs but in many cases you can work around those limits.

It was not until recently that I found a good article about naming conventions. As I created more repositories and the complexity of my projects increased, I become much more cognizant of the importance of being organized. There are many different things that can be done to keep a project orderly but here are some specifics related to Git.

I have to say that I have really enjoyed learning git. Every project I do, I learn to appreciate it a bit more. Part of my interest is for a completely oddball reason - I like using terminal since it fits well in my workflow. I can have a prototype of an idea created that shows basic functionality of the initial goals. This is my proof of concept.

Even though I have spent a significant amount of time over the past two years learning and teaching data science concepts, I find that I am often using other skillsets to solve problems. As engineer, many of the tasks involve optimizing processes and predicting events from a theoretical perspective. Machine learning is very helpful for repetitive tasks or determining human behavior since data exists where this information can be revealed but finding data of optimized processes or rare events is very sparse. My go to skillset is often simulations, algorithms, and linear programming.

Regression is not a topic that I originally spent much time thinking about but as I started to get more involved with analyzing large dataset, I found myself learning more about regression so I coud extract the most out of a dataset. I have noticed one thing as I have worked with data scientiests from different backgrounds.

SQL is a definite requirement for anyone who is interested in doing data work. Here are some basics from work I have done:

Preprocessing might be one of the less appreciated steps in data science. How much time it takes to do is not clear but in general it is one of the lengthier steps in the data preparation process. Part of the ambiguity is associated with what encompasses data preparation. From my experience these are some of the keys steps: