Matmul There are several ways to do matmul in CUDA. Use libraries cutlass cuBlas Implementing your own naive way tiled matmul you can also decide to use shared memory or not It's not the end, yet another way to implement matmul is to use…
Recently I'm learning stuffs with chatGPT. I found that it is exteremly useful to use chatGPT to learn something new. Some Pros: I can dive into codebase first with few prior knowledge. chatgpt isn't pretty good at generating code yet but…
Warning This is not a comprehensive tutorial. It's more a note for myself to write what descisions I made while implementing naive FlashAttention V1. So sadly this also describes my limitation of skills. I already posted an introductory…
I'm extremly poor at thinking about matrices. I've seen many people graphically think and draw matrix strides, multiplications, etc.... Yet as a person who work in ML, I think I should understand matmul, even in high-level concepts. This…
This post is basically a commentary, or more introductory version of this gist . I added some which helped me to understand the article, but more verbosity and confusion may be introduced. Thanks to Kunwar Grover for making a great…
만 서른살이 되었다. 몇년 전부터 유년기의 끝, 청년기의 끝, 뭐 이런 생각들을 해오고 있었는데, 사실 이제부터 중년이 되었다고 봐도 무방한 것 같다. 젊게 살려는 사람들이 많고, 그런 분위기가 아무래도 있지만 나는 왠지 30부턴 좀체 중년에 가깝다(혹은 이미 중년이라)고 자꾸만 생각하게 된다. 이 나이가 될 때까지 도무지 이뤄낸 것이 없고 잘 하는 것도 없지만, 어쨌건 이 나이가 되어버렸다. 갑자기 변할 수는 없겠지만…