In this course I’ve learned about new tests, such as bootstrapping. The principle of generating more samples in order to test to your own sample in this generated distribution was strange at first but after some time I saw the potentials of this strategy. As an extra to more tests I refreshed my R knowledge which will be handy in the rest of the AI bachelor.

The downside of following this course, is that I now know that doing research is hard work and costs lots of time and resources. This makes it not very attractive but it is super important work so I will do my best and put all the new knowledge to work and hopefully contribute in the near future to the research field of AI.


One important thing that I learned in this course was about experiment design. I now know that it’s very important to carefully construct your experiment, and to be very cautious about effects that you do not want, such as those caused by noise and confounding variables.

What we also learned during this course is how a lot of statistical tests actually work. The weekly presentations were really helpful for us to learn the math and reasoning behind these tests. During our statistics course earlier this year, we mostly learned what commands to input into R and interpret the output(especially the p-value), but during this course we gained a deeper insight into what these tests actually mean. We also learned to be more suspicious about certain ‘given’ aspects of statistics that we just accepted in our statistics course, such as why we have a standard alpha of 0.05 and how we should interpret a confidence interval.

Thinking about an imaginary bachalor project was also pretty helpful. It was pretty hard for me to think of a good topic, but it was still a good way to start thinking about it, since we already have to do our bachalor project next year. Commenting on someone else’s imaginary bachalor project was useful as it forced us to think more carefully about how we have to design an experiment.

Lastly, we learned about Bayesian statistics. I found this topic particularly interesting, as it completely changes the conventional way to carry out your statistical tests. I look forward to learning more about Bayesian concepts during other courses, such as machine learning courses.


As I began following this course, I did not know what to expect. As it is in continuation to the statistics course, I expected something similar. Now I figured out what the difference is, and what this course has taught me what the statistics course didn’t.

I learnt to interpret results obtained by various tests. This might look like a general statement, but (for me) there is a big difference between computing and interpreting the results. It will be useful for my bachelor project to be able to compute the results, and then interpret them. I also learnt more about different tests by other students’ presentations.

Another conclusion I can make, is that statistics is hard. It is hard to decide what statistical test to use, also how to use, and when you finally get results, how to interpret it. it This course has cost me a lot of time, but I can say it is worth it. I am more familiar with R and its functions and I hope I can use this new information for my bachelor project.

Reflection on the course

Following the Research Methods course has most importantly taught me a more detailed understanding of statistics, and how it is used in actual research. Learning about statistical tests in general causes you to understand how the tests are performed, what kind of data requires what kind of tests, and how to interpret the results. The Research Methods course expands on this by teaching not only how tests are used in research, but how certain results can be related to the actual research, and what obtaining a certain result means in terms of the research itself.

Additionally, the course teaches not to put blind trust in the statistical tests you perform, to operate with a healthy dose of skepticism. By this I mean that we are taught not to take anything purely at face value, but to be inquisitive regarding the underlying data, and to be thorough in the testing. A statistical test may indicate a significant effect, but if the underlying data contains spurious effects or other faults, the result may not be valid.

This critical thinking is further enhanced by the comparison that is offered between the conventional statistics that we are taught and that are used extensively in research, and the Bayesian statistics alternatives (as presented by Eric-Jan Wagemakers as well). In conclusion, I think the most important thing taught by the Research Methods course is to be critical of your statistics, and to do everything you can to ensure that the test you are using are correct for your data, that your data is as reliable as possible, and to thoroughly consider what you can actually conclude from your data.

Reflection on the course

What I learned, as did I from the previous 4 attempts to this course, is that research (methods) isn’t my cup of tea. Even though I put quite a bit of effort into it this year, the grades just don’t seem to keep an even trend with that. Obviously this is too bad but some studying for the exam will hopefully get me through it.

The course itself and its contents taught me on a lot of subjects I hadn’t studied or looked at before. For me statistics was just looking at a number that R spit out so that I could conclude something for my assignment, whilst I now know and understand a lot more of the background of most tests.  My view on my study hasn’t changed after this course, maybe it even strengthened me in my beliefs that I don’t want to continue in the research area of AI.

Reflection: bayes is neat

I learned a lot in this course, how to better work with R, got a better understating of which t.tests I should, what tests I can run on certain types of data, how to compare models enc. But I think what I found most interesting in this course was bayes and not just because a cool lecturer with a southpark example came along who told me if I knew bayes I would not need any other form of statistical test. I like bayes because it is  a widely used plausibility measuring tool, used in many field including of course AI. It is used in speech recognition, decision making algorithms and I never truly got the grasp of it, I was shown formulas but did not really understand what they meant. By comparing hypothesis as a factor you can also calculate risk, you can know by how much the data supports one outcome versus another. Bayes is neat because it does not measure probability, it measures plausibly. The true purpose of bayes is to predict the world from what you know, as soon as you have obtained new knowledge you can predict the world better. I like bayes that is all.

Statistical enlightenment thanks to randomization tests

There are a lots of things that I learned during this course, but the most notable thing may be that I finally managed to wrap my head around classical hypothesis testing with null hypotheses, p-values and the like. Unlike the vast majority of aspiring statisticians and researchers, my introductory statistics course foucussed soley on Bayesian statistics. The book we used during that course made a really big deal about the difference, but I didn’t understand the other camp at all: the ‘frequentist’ philosophy struck me as unscientific, its method of confirming and rejecting hypotheses confused me and the frequentist statistical tests were like magic to me.

I do get it now, I really do. P(data|H0) i.e. the probability of the data given the null hypothesis (instead of P(H|data), i.e. the probability of some hypothesis given the data). When the data is unlikely given the null, we may choose to reject the latter.

I have to admit one thing though. The most famous frequentist tests: the t-tests and (more generally) analysis of variance tests are still mostly like magic to me. Sure, I know how to call t.test or aov from R, but what do these functions do exactly? How do these tests work? I tried to read more about it, to really understand these tests, but it seems like the more research I do, the deeper I dig, the more formulas, concepts and magical statistics I encounter that only serve to confuse me further.

Everything changed when we were introduced to randomization tests. You want to see if two populations differ? Well, just randomly shuffle all the numbers around a couple of hunderd times and calculate the means of each group. Brilliant! No t-statistic, normalization of the data, weird counter-intuitive prerequisites or formulas to remember – a simple algorithm is all it takes to compare two, or just any amount of groups!

So, you could say that I have gained ‘statistical enlightenment’ by being introduced to a statistical test that just makes sense to me. Once I start doing hard-core emperical research myself I may start to appriciate all the other statistical methods introduced in this course. Until then, I know what test I will use for quick and dirty (but no less valid) statistics.